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Questions  regarding  the  usefulness  of  the  Naval  Postgraduate 
School's  Student  Opinion  Form  (SOF)  as  a  device  to  measure 
teaching  effectiveness  have  prompted  this  research.   The 
possibility  that  the  SOF  may  weigh  heavily  in  pay,  promotion, 
and  tenure  decisions  is  cause  for  research  into  its  validity 
and  reliability  as  an  evaluation  instrument.   The  first  of 
three  separate  studies  described  here  consists  of  an  analysis 
of  a  questionnaire  distributed  to  all  teaching  professors  in 
the  NPS  Administrative  Sciences  Department.   The  second  study 
concerns  a  questionnaire  completed  by  258  Administrative 
Sciences  students,  and  the  third  study  considers  the  responses 
of  560  students  to  four  supplementary  items  added  to  the  SOF. 
The  results  indicate  that  neither  students  nor  faculty 
members  feel  strongly  that  SOF's  actually  measure  or  improve 
teaching  effectiveness,  that  a  large  part  of  the  variation 
in  SOF  ratings  is  attributable  to  factors  other  than  a  pro- 
fessor's teaching  quality  and,  finally,  that  a  student's 
anticipated  course  grade  or  cumulative  grade  point  average 
has  little  correlation  with  the  SOF  ratings  given  the 
professor. 
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I.   INTRODUCTION 

A.   MOTIVATION 

In  recent  years,  a  great  deal  of  interest  has  been 
expressed  about  evaluating  the  quality  of  instruction. 
Reasons  for  this  interest  appear  to  include  decreasing  college 
enrollments,  increasingly  demanding  educational  goals,  and 
a  growing  emphasis  on  instructional  accountability.   Since 
1975,  the  Student  Opinion  Form  (SOF)  has  been  used  at  the 
Naval  Postgraduate  School  as  a  tool  to  measure  the  teaching 
effectiveness  of  instructors.   Student-faculty  evaluations 
(SFE's)  of  instructor  effectiveness  have  become  widely 
accepted  in  the  academic  world  as  helpful  indicators  of 
performance,  but  there  has  been  considerable  controversy 
regarding  their  validity  for  use  in  pay,  promotion,  and 
tenure  decisions.   In  addition  to  the  questionable  validity 
of  SFE's,  another  common  criticism  is  that  they  are  biased 
by  variables  that  are  unrelated  to  teaching  effectiveness. 
Frequently,  however,  computer  formatted  students'  evaluations 
are  the  only  form  of  faculty  evaluation  used,  mainly  because 
of  ease  of  administration  and  processing. 

Many  college  faculty  fear  that  too  much  emphasis  on 
student -faculty  evaluations  may  lead  to  manipulative  and 
other  corruptive  practices  that  would  have  a  negative  impact 
on  the  quality  of  education  and  on  the  academic  community 


in  general.   These  concerns  are  certainly  not  new  to  the 
academic  world.   In  fact,  these  problems  have  plagued 
educators  as  far  back  as  medieval  times.   Historical  writings 
about  medieval  European  universities  discuss  the  restrictive 
methods  resorted  to  in  an  attempt  to  prevent  these  poten- 
tially corruptive  practices.   Professors  were  paid  a  "col- 
lecta"  or  fee  by  each  student  to  teach  an  agreed-upon  amount 
of  material  by  a  specified  date.   A  group  of  students  was 
appointed  by  the  rector  to  report  negligent  professors  who 
had  not  fulfilled  their  contractual  agreement.   Based  upon 
this  group's  evaluation,  a  fine  would  be  imposed  upon  the 
professors  for  each  day  they  had  fallen  behind  in  their 
teachings.   Although  this  may  be  viewed  as  an  extreme  response 
to  the  problem,  many  educators  today  question  the  ultimate 
impact  of  student  evaluations  on  teaching  effectiveness. 
These  concerns,  as  well  as  the  potential  benefits  to  be 
gained  from  student-faculty  evaluations,  have  triggered 
numerous  studies. 

B.   RESEARCH 

Research  interest  in  student-faculty  evaluation  (SFE) 
has  increased  significantly  in  recent  years.   This  concen- 
trated attention  not  only  is  due  to  student  interest  but 
also  seems  to  be  a  function  of  the  increased  use  of  SFE 
in  the  determination  of  faculty  promotions  and  salaries. 
In  fact,  the  agencies  that  govern  public  colleges  and 
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universities  in  some  states  require  that  SFE  be  employed 
as  one  component  in  pay,  promotion,  and  tenure  decisions. 
This  requirement  is  apparently  based  on  the  premise  that 
SFE  provides  a  precise  and  quantifiable  evaluation  of  in- 
structors.  However,  the  assignment  of  numbers  to  instructor/ 
course  attributes  does  not  automatically  confer  reliability 
or  validity  upon  the  rating  system  as  a  whole.   There  have 
been  numerous  studies  with  extremely  inconsistent  results 
as  to  the  relationship  between  SFE  and  teaching  effective- 
ness (as  measured  by  amount  learned).  [Ref.  1]   Studies  that 
resulted  in  both  highly  positive  [Ref.  2]  and  highly  negative 
correlations  [Ref.  3]  have  been  reported;  however,  the  majority 
of  these  studies  yielded  correlations  that  did  not  differ 
significantly  from  zero  [Ref.  4]  . 

The  major  aim  of  student-rating  proponents  is  basically 
to  obtain  more  valid,  reliable,  and  effective  means  of 
incorporating  the  evaluation  of  teaching  into  advancement  pro- 
cedures that  might  otherwise  be  available.   These  proponents 
also  feel  that  there  are  many  benefits  to  be  gained  from 
these  evaluations.   Specifically,  Costin,  Greenough,  and 
Menges,  in  an  extensive  review  of  reliability,  validity,  and 
usefulness  of  student  ratings,  state  that: 

1.   Such  ratings  could  provide  feedback  which  the  in- 
structor might  not  be  able  to  elicit  from  students  on 
a  face-to-face  basis.   (This  information  alone,  with 
no  sanctions  contingent,  could  improve  teaching.) 
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2.  They  could  provide  departmental  and  college-wide 
norms  against  which  individual  faculty  ratings  could 
be  judged. 

3.  They  could  provide  a  way  in  which  a  faculty  member 
could,  if  he  desired,  demonstrate  his  undergraduate 
teaching  effectiveness  to  those  who  have  expressed  an 
interest  in  evaluating  this  parameter  for  salary 
increase . 

4.  They  could  provide  information  to  the  department  and 
college  on  areas  of  relative  strength  or  weakness  in 
undergraduate  teaching,  suggest  directions  for  the 
development  of  new  courses  or  programs,  and  provide 
evaluative  information  and  norms  on  the  various  new 
programs  which  are  implemented. 

5.  They  could  provide  the  student  with  a  source  of 
information  to  aid  him  in  the  selection  of  courses. 
[Ref.  5] 

It  should  be  noted  that  these  benefits  can  exist  only  to  the 
extent  that  student-faculty  evaluations  portray  accurate  and 
valid  appraisals  of  classroom  instruction. 

Maslow  and  Zimmerman,  when  asking  students  to  make 
global  ratings  of  teachers'  effectiveness,  provided  the  fol- 
lowing definitions  to  aid  students  in  making  their  ratings: 

.  .  .  A  person  deserving  the  highest  possible  rating, 
as  a  teacher,  was  described  as  one  who  is  both  capable 
and  efficient,  who  loves  his  job  and  manages  to  inspire 
his  students,  who  is  himself  inspired  with  his  work, 
who  is  talented,  and  who  not  only  respects  and  appre- 
ciates his  students,  but  also  has  good  relations  with 
them.   The  highest  rating  as  a  personality  was  to  be 
given  to  a  very  healthy,  well  integrated  person,  sub- 
jectively at  ease  with  himself,  happy  or  content,  using 
all  his  constructive  capacities,  enjoying  life  without 
neurotic  or  psychotic  maladjustments.  [Ref.  6] 

Though  Maslow  and  Zimmerman  found  a  0.76  correlation  between 

students'  ratings  of  "good  teaching"  and  "good  personality," 

both  personality  and  ability  were  so  vaguely  defined  as  to 
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cause  difficulty  in  interpreting  the  results  with  any 
precision.  [Ref.  7] 

It  cannot  be  denied  that  SFE  is  influenced  by  many  non- 
instructional  variables,  which,  more  than  likely,  account  for 
the  inconsistent  findings,  as  noted  above.   Variables  such 
as  class  size,  course  level,  presentational  style,  actual  or 
expected  grade,  and  student  group  are  all  likely  major  deter- 
minants of  SFE.   For  instance,  an  interesting  example  of  the 
effect  of  presentational  style  and  format  was  demonstrated 
at  one  school  in  which  a  professional  actor  was  hired  to 
deliver  a  series  of  lectures  to  a  large  student  group. 
Because  of  his  enthusiastic,  humorous,  and  personable  manner, 
he  received  extremely  high  ratings;  though  entertaining, 
educational  content  of  his  presentation  was  low  or  totally 
absent.  [Ref.  8]   The  favorable  rating  in  this  case  can  be 
considered  to  be  a  function  of  superficial  popularity  or 
what  is  sometimes  referred  to  as  the  "popularity  halo." 

In  another  study.  Brown  (1976)  found  that  student  grades 
accounted  for  approximately  nine  percent  of  the  variance 
among  student  ratings  (on  the  average  across  classes) .   It 
was  his  interpretation  that,  as  a  result  of  this  small  but 
positive  relationship  between  student  grades  and  student 
ratings,  professors  can  technically  "buy"  high  student  ratings 
by  giving  their  students  high  grades.  [Ref.  9]   This  is  a 
very  popular  interpretation  of  the  frequent  low  correlations 
and  poor  regression  results  that  are  obtained.   Conversely, 
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Voeks  and  French  have  found  in  their  studies  that  "high  ratings 
cannot  be  'bought'  by  giving  high  grades,  nor  are  they  lost 
by  giving  low  grades."  [Ref.  10]   This  finding  would  imply 
that  college  students  have  greater  objectivity  and  less 
superficial  value  systems  than  they  have  been  given  credit 
for  by  others. 

Howard  and  Maxwell  (1980)  have  proposed  three  models 
that  attempt  to  bridge  the  gap  between  these  two  schools  of 
thought.  [Ref.  11]   Their  first  model  (Figure  1)  very  simply 
states  that  students'  evaluations  are  directly  dependent  on 
their  expected  or  actual  grades. 


ACTUAL 

OR 

EXPECTED  GRADE 


STUDENT 
EVALUATION 


Figure  1.   Grading  Leniency  Bias  Model 

Their  second  model  (Figure  2)  suggests  that  greater  teaching 
effectiveness  causes  improved  student  performance,  which 
consequently  results  in  both  higher  student  grades  and 
student  evaluations.   Howard  and  Maxwell's  final  model 
(Figure  3)  proposes  that  greater  student  motivation  causes 
better  student  performance,  resulting  in  higher  student  grades 
and  higher  student  evaluations. 
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Figure  2.   Teaching  Effectiveness  Model 
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Figure  3.   Student  Characteristics  Model 
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Figures  2  and  3  both  exhibit  a  positive  relationship 
between  student  grades  and  student  evaluations.   However, 
this  association  is  interpreted  as  a  logical  result  of  student 
performance  and  not  bias.   As  can  be  seen,  there  is  little 
agreement  among  researchers  about  the  effects  of  student 
grades  on  student  evaluations,  and  even  concerning  the  inter- 
pretation of  the  research  results. 

Several  researchers  have  also  focused  their  studies  in  the 
direction  of  instructor  personality  traits.   Results  in  this 
field  reveal  that  "overall  effectiveness"  of  teaching,  as 
perceived  by  students,  seems  to  be  positively  related  to 
instructor  characteristics  (e.g.,  imagination,  intelligence, 
emotional  stability,  enthusiasm).   For  example,  Hildebrand 
and  Wilson  conducted  a  highly  acclaimed  research  project  of 
this  kind.   Their  main  goal  was  to  develop  a  reliable  instru- 
ment that  would  provide  a  basis  for  evaluating  teaching  that 
could  also  be  incorporated  into  advancement  procedures. 
Their  report  discusses  the  development  of  three  forms  of 
varying  lengths  that  could  be  used  as  teaching  evaluation 
instruments  for  the  California  State  University  system.   Both 
student  and  faculty  characterizations  of  effective  teaching 
were  assessed,  but  the  final  recommended  instrument  relied 
heavily  on  the  students'  characterization.   Five  scales 
(teacher  description  scales) ,  established  from  this  instru- 
ment via  factor  analysis,  have  the  following  conceptual 
interpretations : 
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1.  Analytic/Synthetic  Approach- -scholarship ,  with 
emphasis  on  breadth,  analytic  ability,  and  conceptual 
understanding. 

2.  Organization/Clarity--skill  at  presentation,  but  is 
subject-related,  not  student-related,  and  not  merely 
concerned  with  rhetorical  skills. 

3.  Instructor- Individual  Student  Interaction- -rapport 
with  the  class  as  a  whole,  sensitivity  to  class  response, 
and  skill  at  securing  active  class  participation. 

4.  Instructor-Individual  Student  Interaction- -mutual 
respect  and  rapport  between  the  instructor  and  the 
individual  student. 

5.  Dynamism/Enthusiasm- -flair  and  infectious  enthusiasm 
that  comes  with  confidence,  excitement  for  the  subject, 
and  pleasure  in  teaching.  [Ref.  12] 

These  five  scales  have  become  widely  acknowledged  and  several 
researchers  have  based  their  studies  on  them. 

Standard  characteristics  currently  found  on  over  twenty- 
eight  student-faculty  evaluations  were  reviewed  in  this 
study.   The  following  factors  seemed  to  be  the  most  relevant 
and  popular:   course/lecture  organization,  instructor  atti- 
tude toward  student  understanding,  instruction  preparation, 
clarity  of  explanation,  instructor  knowledge,  instructor 
evaluation  of  student,  ability  to  teach  at  appropriate  level, 
instructor  attitude/enthusiasm  toward  course,  instructor 
attitude  toward  questions,  instructor  control  of  class  time, 
instructor  availability,  instructor  ability  to  evoke  interest, 
clarity  of  course  objectives,  instructor  attitude  toward 
student  progress,  rapport  with  students,  course  workload, 
and,  finally,  relations  of  subject  matter  to  real  world 
applications.   There  also  have  been  studies  that  numerically 


17 


rank,  by  importance,  the  above  factors  by  polling  student 
groups.   The  majority  of  these  studies  found  course  organi- 
zation to  be  the  most  important  factor. 

Other  studies  have  taken  a  slightly  different  approach 
and  have  attempted  to  determine  a  set  of  mutually  exclusive 
factors  that  are  relevant  to  satisfaction  or  dissatisfaction 
with  teacher  performance.   For  example,  Elster  et  al.  used 
the  critical  incident  technique  to  determine  the  variables 
that  lead  to  students'  satisfaction  with  instructors  at  the 
Naval  Postgraduate  School.   The  students  (over  250)  were  asked 
to  identify  a  time  when  they  were  especially  satisfied  or 
dissatisfied  with  a  teacher  and  to  describe  what  had  led 
them  to  feel  that  way.   A  set  of  collectively  exhaustive 
and  mutually  exclusive  content  categories  were  developed. 
The  category  labeled  "organization  and  preparation"  was  the 
most  frequently  appearing  response,  followed  by  "instructor's 
traits"  and  "instructor's  classroom  or  presentation  tech- 
niques." The  "instructor's  knowledge  of  the  subject"  and 
"evaluation  of  students  in  general"  were  also  categories 
that  were  frequently  mentioned.  [Ref.  13]. 

Similarly,  Gadzella  (1968)  asked  a  randomly  selected 
sample  of  students  to  state  their  opinions  of  an  "ideal 
professor."  This  study  found  that  the  four  most  important 
criteria  were  knowledge  of  subject,  interest  in  subject, 
flexibility,  and  preparation.  [Ref.  14]. 
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Costin  (1968)  asked  over  200  graduate  and  undergraduate 
students  from  three  different  universities  to  rate  statements 
regarding  the  classroom  behavior  of  the  "best  lecturer" 
that  they  had  ever  encountered.   A  four-point  integer  scale 
ranging  from  (4)  "almost  always  occurred"  to  (1)  "almost 
never  occurred"  was  used.   Items  such  as  "acted  interested 
in  the  material,"  "was  well  prepared,"  "used  relevant  examples," 
"followed  a  logical  sequence  of  thought,"  and  "explained 
clearly"  received  mean  score  ratings  above  3.5.  [Ref.  15] 

It  has  been  disputed  that  students'  judgments  of 
desirable  criteria  of  teaching  effectiveness  many  times  are 
immature  and  inaccurate  and  lack  long-term  perspective.   This 
point  has  been  discredited  on  numerous  occasions.   For  example, 
Drucker  and  Remmers  asked  college  alumni  what  they  thought 
the  most  important  qualities  of  a  good  instructor  were,  and 
compared  the  answers  with  current  undergraduates'  views. 
They  found  that  both  groups  agreed  identically  on  the 
ranking  of  these  criteria: 

a.  adequacy  of  preparation 

b.  interest 

c.  stimulation  of  intellectual  curiosity 

d.  "progressive"  attitude 

In  addition,  they  found  that  student  ratings  correlated 
positively  with  alumni  ratings  of  the  same  instructors 
[Ref.  16]. 
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In  summary,  much  research  has  been  completed  in  the  above 
areas  of  student-faculty  evaluation.   Although  researchers' 
approaches  may  differ,  they  are  all,  in  essence,  attempting 
to  determine  the  most  reliable,  valid,  and  appropriate 
method  of  student-faculty  evaluation. 

C.   SOF  HISTORY  AND  DEVELOPMENT 

The  Student  Opinion  Form  (SOF) ,  which  is  presently  used 
at  the  Naval  Postgraduate  School,  was  developed  locally  and 
instituted  in  June  of  1975.   This  particular  form  replaced 
the  Student  Instruction  Report  (SIR) ,  which  was  a  widely 
used  form  obtained  from  the  Educational  Testing  Service  (ETS) 
at  Princeton.   The  SIR  elicited  students'  judgments  of  such 
aspects  as  the  organization  of  the  course,  the  pace  of  the 
course,  the  instructor's  helpfulness  or  availability  to 
students,  and  the  clarity  of  objectives  and  presentations. 
The  SIR  contained  39  questions  while  the  presently-used  SOF 
contains  16  questions  as  well  as  a  provision  for  written 
comments.   The  free-form  comments  afford  the  student  a  means 
of  additional  communication  and  an  opportunity  to  convey 
specific  feedback  to  the  instructor  (which  is  available  only 
to  the  instructor) .   The  SOF  was  adapted  from  the  technique 
of  Hildebrand  and  Wilson  (1971)  and  formatted  after  SIR. 
More  specifically,  seven  of  the  questions  on  the  current  SOF 
were  taken  directly  from  the  SIR  form  and  the  remaining  ques- 
tions were  added  by  an  appointed  committee  of  NPS  professors. 
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The  present  SOF  system  requires  that  these  SOF  forms 
be  distributed  at  the  end  of  each  term  and  that  all  students 
individually  fill  out  the  sixteen-item  questionnaire  for 
each  course  that  they  are  enrolled  in.   Items  one  through 
eleven  are  behavioral  in  nature  while  items  twelve  through 
sixteen  are  "overall"  ratings.   These  questionnaire  items 
are:   (1)  The  course  was  well  organized;  (2)  Time  in  class 
was  spent  effectively;  (3)  The  instructor  seemed  to  know  when 
the  students  did  not  understand  the  material;  (4)  Difficult 
concepts  were  made  understandable;  (5)  I  had  confidence  in 
the  instructor's  knowledge  of  the  subject;  (6)  I  felt  free 
to  ask  questions;  (7)  The  instructor  was  prepared  for  class; 
(8)  The  instructor's  objectives  for  the  course  have  been 
made  clear;  (9)  The  instructor  made  this  course  a  worthwhile 
learning  experience;  (10)  The  instructor  stimulated  my 
interest  in  the  subject  area;  (11)  The  instructor  cared 
about  student  progress  and  did  his  share  in  helping  us  learn; 
(12)  Overall  rating  of  the  instructor;  (13)  Overall  rating 
of  the  course;  (14)  Overall  rating  of  the  textbook(s); 

(15)  Overall  rating  of  the  quality  of  the  exams;  and 

(16)  Overall  rating  of  the  laboratories.   All  of  these 
items  are  scored  on  an  integer  scale  from  zero  to  five.   The 
response  options  for  the  behavioral  items  are:   no  comment, 
strongly  disagree,  disagree,  no  strong  opinion,  agree,  and 
strongly  agree,  respectively;  the  response  scale  for  the 
overall  items  is  as  follows:   N/A,  poor,  fair,  average, 
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excellent,  and  outstanding,  respectively.   Appendix  A  shows 
an  actual  SOF  used  in  this  study  for  analysis  purposes. 

The  NPS  SOF's  were  processed  in-house  for  several  years 
via  an  optical  scanning  machine.   Numerous  mechanical  break- 
downs of  the  op-scanner,  as  well  as  machine  processing 
inaccuracy,  resulted  in  a  great  need  for  a  more  flexible  and 
accurate  system.   In  the  spring  of  1983,  processing  was 
delegated  to  a  civilian  contractor,  McGraw-Hill,  which 
enabled  the  Naval  Postgraduate  School  to  receive  results 
approximately  one  week  following  submission.   Processing 
costs  of  the  SOF  run  approximately  $.25  per  page.   NPS  sub- 
mits between  6000  and  7000  forms  per  quarter  for  optical 
scanning  at  McGrawHill. 

Student-faculty  evaluations  at  NPS  were  initially 
intended  to  be  used  strictly  as  feedback  for  the  individual 
professors  to  aid  in  the  improvement  of  instruction.   They 
have  since  evolved  into  a  supportive  evaluation  tool  to  be 
used  by  faculty  administrators  as  an  overall  aid  in  pay, 
promotion,  and  tenure  decisions.   This  evolution  into  an 
administrative  use  has  created  considerable  controversy. 

Zelby  states  that  one  major  cause  for  the  growth  of  the 
student- faculty  evaluation  controversy  is  the  "trend  toward 
formal,  quantitative  use  of  the  results  of  the  evaluations 
in  determinations  of  faculty  promotions  and  salaries."  He 
states  further  that  there  are  three  reasons  for  this  trend: 
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(1)  SFE  provides  documented,  precise,  empirical  evalua- 
tions of  instructors;  (2)  it  tends,  thereby,  to  relieve 
academic  administrators  from  the  responsibility  of 
exercising  judgment  about  teaching  performance  and 
ability;  and  (3)  it  tends  to  constitute  proof  that 
something,  indeed,  is  being  done  to  improve  teaching. 
[Ref.  17] 

In  1977  and  1978  there  seemed  to  be  a  growing  interest 
on  the  part  of  NPS  faculty  and  students  in  the  Student 
Opinion  Form.   In  April  of  1977,  the  NPS  Scholarship 
Committee  submitted  a  memorandum  to  the  Faculty  Council 
recommending,  among  other  things,  the  initiation  of  a  statis- 
tical research  project  to  determine  the  significance  of  SOF 
scores.   In  addition,  the  Student  Council  took  the  initiative 
to  prepare  a  SOF  information  sheet  for  dissemination  to  the 
student  body  based  on  information  acquired  from  NPS  faculty 
administrators.   Appendix  B  contains  a  copy  of  the  Student 
Council's  memorandum.   Interest  was  so  great  as  to  evoke 
the  development  of  a  proposed  new  Student  Opinion  Form.   This 
alternative  form  was  never  instituted  at  NPS,  though  it 
offered  many  innovative  ideas. 

When  the  SOF  was  initially  developed,  strict  guidelines 
were  outlined  for  its  administration.   Unfortunately,  in  the 
past  ten  years  there  has  been  a  decline  in  the  emphasis 
placed  on  the  careful  adherence  to  these  guidelines,  result- 
ing in  the  present  lack  of  consistency  in  SOF  administration 
among  course  segments. 

The  initial  guidelines  required  the  SOF's  to  be  com- 
pleted within  the  last  two  weeks  of  the  quarter,  prior  to 
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finals,  and  during  class  time.   One  SOF  was  to  be  filled  out, 
in  the  absence  of  the  professor,  by  each  student  for  each 
class  segment  he  or  she  took  for  course  credit.   Specific 
instructions  for  completion  of  the  SOF  were  provided  on 
each  form  (e.g.,  soft  black  pencil,  no  ink).   The  SOF  forms 
were  to  be  disseminated  by  the  professor  and  collected  by 
the  appointed  section  leader  at  the  end  of  the  allotted 
time.   Once  collected,  the  SOF's  were  to  be  placed  in  an 
envelope  and  immediately  submitted  to  the  department  office. 
Neither  the  professor  nor  the  section  leader  was  to  review 
the  completed  forms.   Once  the  processing  of  the  SOF  was 
completed,  summary  statistical  data  were  provided  to  the 
department  chairman  and  the  actual  SOF's  were  returned  to 
the  professor. 

The  original  guidelines  are  still  in  effect;  however, 
because  of  inadequate  guidance  and  lax  enforcement,  little 
consistency  in  SOF  administration  exists  in  the  system  today. 
It  is  not  unusual  for  the  SOF's  to  be  filled  out  at  home, 
completed  at  a  later  date  (sometimes  well  into  the  following 
quarter),  or  never  completed  at  all.   This  problem  has  been 
compounded  by  the  frequent  complaints  of  insufficient  time 
being  provided  to  fill  out  the  SOF's.   There  are  also 
numerous  instances  of  the  professor  remaining  in  the  class- 
room and  wandering  amongst  the  students  during  SOF  completion, 
Another  problem  is  that  students  tend  not  to  read  the 


24 


instructions  and,  hence,  to  fill  out  the  computer  forms 

incorrectly.   In  addition,  extraneous  markings  or  comments 

are  frequently  made  in  areas  other  than  those  designated. 

It  also  has  become  commonplace  for  the  section  leaders 

and  other  student  members  to  review  the  completed  SOF's 

in  their  entirety  prior  to  submission.   Because  of  the  lack 

of  specific  and  comprehensive  instructions  or  other  guidance, 

many  of  the  above  infractions  contribute  to  further  poor 

compliance,  as  can  be  seen  in  the  mean  SOF  return  rate.   As 

an  example,  the  SOF  percentage  return  rate  in  the  Administrative 

Sciences  Department  for  the  past  eight  years  has  averaged  761. 

(See  Appendix  C.)   Despite  a  slightly  increasing  return  rate 

trend  over  this  period,  there  is  still  considerable  apathy 

among  students  toward  SOF's.   Unless  greater  emphasis  is 

placed  on  the  careful  completion  of  SOF  forms  and  new 

interest  rekindled  in  their  validity  as  an  evaluation  tool, 

their  usefulness  if  not  their  use  will  continue  to  decline 

until  they  serve  no  further  purpose. 

In  addition  to  these  problems,  there  are  several  other 
concerns  with  the  processing  and  utilization  of  the  SOF. 
As  was  previously  stated,  processing  time  at  McGraw-Hill  re- 
quires approximately  one  week;  therefore,  it  was  anticipated 
that  the  completed  SOF  data  would  be  returned  to  the  faculty 
during  the  first  week  of  the  following  term.   Unfortunately, 
a  one-and-a-half  to  two-month  turnaround  time  has  become 
reality.   This  delay  is  primarily  due  to  careless  completion 
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of  SOF's,  thus  requiring  extensive  editing.   At  present, 
limited  available  resources  allow  for  only  one  editor  to 
make  these  corrections.   Since  nearly  20%  of  the  6000  com- 
pleted SOF's  require  editing,  this  job  generally  exceeds 
one  month.   In  essence,  it  was  unrealistic  to  expect  a  one- 
to  two-week  turnaround  time;  however,  this  tremendous  edit- 
ing problem  was  never  expected. 

By  the  time  the  evaluations  are  available  to  the  pro- 
fessors, the  school  is  well  into  the  following  term.   The 
SOF  information  by  this  time  may  be  virtually  irrelevant, 
since  the  instructors  are  most  likely  teaching  a  different 
student  group  and  possibly  a  different  course.   In  any  case, 
feedback  received  at  the  beginning  of  the  quarter  would 
allow  sufficient  time  to  make  changes  in  course  structure, 
exams,  or  even  possibly  teaching  approach. 

D.   DIRECTION  OF  RESEARCH 

The  review  of  much  of  the  previous  research  in  the  area 
of  SFE,  as  well  as  the  history  of  NPS's  own  SOF,  led  to  an 
inquiry  into  several  areas  related  to  SFE.   Over  the  past 
nine  years,  it  appears  that  interest  in  SFE  has  waned. 
During  the  time  between  1965  and  1975,  it  was  evident  that 
there  was  a  burgeoning  concern  regarding  the  quality  of 
college  and  university  teaching.   Prior  to  this  time,  research 
and  publication  were  considered  to  be  of  primary  importance  in 
the  evaluation  of  a  professor's  capabilities. 
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Historically,  every  generation  of  students  has  been  hungry 
for  good  teachers,  but  the  current  generation  has  been  the 
first  in  this  country  to  mount  an  organized  attack  on  what 
they  charge  has  been  the  Establishment's  lack  of  concern 
for  the  poor  quality  of  college  instruction.  [Ref.  18] 

In  order  for  quality  of  teaching  to  improve,  a  reliable  method 
for  measuring  teaching  effectiveness  must  be  incorporated 
into  the  educational  system. 

The  situation  just  described  prompted  the  authors  of  this 
paper  to  address  these  issues  pertaining  to  SFE  at  the  Naval 
Postgraduate  School.   Three  individual  studies  were  pursued 
in  this  area.   All  data  collected  were  restricted  to  the  NPS 
Administrative  Sciences  Department.   The  first  study  con- 
cerned a  professor  questionnaire  on  the  NPS  Student  Opinion 
Form  that  was  distributed  to  fifty  instructors  who  were 
then  teaching  at  least  one  course.   The  second  involved  a 
student  questionnaire  on  the  SOP  that  was  disseminated 
to  258  students.   Finally,  in  the  third  study,  four  addi- 
tional questions  were  added  to  the  currently-used  NPS  SOF 
for  the  purpose  of  gathering  further  SOF  information. 
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II.   PROFESSOR  QUESTIONNAIRE 

A.   MOTIVATION 

The  decision  to  design  and  administer  a  professor  ques- 
tionnaire was  primarily  motivated  by  candid  comments  that 
emerged  from  informal  conversations  with  professors  from 
different  departments  at  the  Naval  Postgraduate  School.   In 
addition,  myths  and  rumors  regarding  the  use  and  purpose  of 
Student  Opinion  Forms  have  abounded  among  students  and 
faculty.   There  is  as  much  variation  in  opinion  by  the  pro- 
fessors as  there  is  among  students  on  the  value  of  the  SOF 
as  a  feedback  and  evaluation  tool.   Hutchison  cautions 
against  the  use  of  student-faculty  evaluations  for  evalu- 
ation purposes  because  the  students  are,  in  essence,  merely 
reporting  perceptions,  not  making  performance  appraisals. 
[Ref.  19]   It  was  our  general  impression  that  many  professors 
seriously  question  the  validity  and  reliability  of  the  SOF, 
and  place  little  credence  in  the  SOF  results, 

A  study  by  Herbert  W.  Marsh  indicates  that  this  skepti- 
cism about  student -faculty  evaluations  is  not  uncommon  among 
faculty  members. 

Faculty  are  concerned  about  teaching  effectiveness, 
even  to  the  extent  of  wanting  it  to  play  a  major  role 
in  administrative  decisions,  but  have  no  confidence 
in  any  measures  of  teaching  effectiveness  - -including 
students'  evaluations  ....  An  important  role  of 
research  in  students'  evaluations,  besides  demon- 
strating their  reliability,  validity,  and  lack  of 
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bias,  is  to  convince  faculty  and  administrators  of 
their  worth.  [Ref.  20] 

For  example,  some  professors  feel  the  senior  class  member  (or 

section  leader)  has  a  great  deal  of  influence  over  the  ratings 

received  by  the  professor.   It  has  been  rumored  that  in  some 

cases  the  class  section  leader  has  organized  meetings  of  the 

class  members  to  arrive  at  a  common  "grade"  for  the  professor's 

SOF.   (This  may  be  true,  especially  if  the  professor  has  not 

lived  up  the  expectations  of  the  class  or,  in  particular,  the 

class  leader.)   Such  collusive  practices  only  serve  to  support 

faculty  reservations  about  the  accuracy  of  the  students' 

evaluations . 

Another  study,  by  Robert  R.  Read  of  the  Naval  Postgraduate 
School,  revealed  that  the  effects  of  two  factors  other  than 
the  professor's  actual  teaching  ability,  student  group  and 
the  particular  course,  enter  into  the  SFE  ratings.   Read 
found  through  analysis  of  variance  that,  of  these  three 
factors,  the  effect  of  the  course  was  of  supreme  importance; 
and,  surprisingly  enough,  the  effect  of  the  student  group 
also  surpassed  the  effect  of  the  professor.   Professor  Read 
concluded  that  Item  12  of  the  NPS  SOF  (overall  rating  of 
instructor)  does  not  measure  what  it  claims  to  measure  because 
of  the  strong  course  and  student-group  components  of  variation 
in  response  to  this  particular  item.  [Ref.  21] 

In  spite  of  this  lack  of  confidence  in  the  validity  of 
student -faculty  evaluations,  they  are  often  the  only 
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available  measure  of  teaching  effectiveness;  and  professors 
and  administrators  are  therefore  led  to  rely  upon  them. 
This  reliance  is  evidenced  by  the  frequent  use  of  student- 
faculty  evaluations  for  pay,  promotion,  and  tenure  purposes. 
Wilson,  Gaff,  and  Bavry  (1970)  surveyed  one  thousand  faculty 
members  from  six  schools  regarding  the  criteria  that  are  used 
by  college  administrators  in  making  advancement  decisions. 
Ninety-two  percent  of  those  surveyed  felt  that  teaching 
effectiveness  should  be  quite  important  or  very  important  in 
promotion  decisions,  although  only  38%  stated  that  teaching 
effectiveness  actually  is  considered  important  to  this  extent 
in  advancement  decisions.   Seventy- two  percent  of  the  faculty 
members  surveyed  felt  that  a  formal  evaluation  procedure 
should  be  instituted  at  their  schools.  [Ref.  22]   Of  course, 
research  and  publications  play  an  important  role  in  these 
decisions;  however,  there  are  no  formal  quantitative  methods 
of  evaluation  used  at  NPS  to  measure  these  particular  com- 
ponents of  professor  performance. 

Although  the  SOF  covers  a  wide  variety  of  items,  there 
is  a  noted  tendency  to  concentrate  on  SOF  Item  12,  the  over- 
all rating  of  the  instructor.   In  particular,  Item  12  is 
relied  upon  by  some  department  chairmen  as  a  "quick  and  dirty" 
overview  of  the  professors'  teaching  capabilities.   Not  unlike 
notification  of  students  who  make  the  Dean's  List,  congratula- 
tory letters  are  sent  to  NPS  faculty  whose  mean  score  on  this 
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particular  item  is  high.   In  fact,  at  least  one  department 
at  the  Naval  Postgraduate  School  rank-orders  its  professors 
based  solely  on  their  mean  Item  12  score.   This  practice  may 
be  a  self-defeating  one  because  of  the  potential  detriment 
to  the  morale  and  self-esteem  of  professors  who  rank  low. 

Such  great  emphasis  on  the  Student  Opinion  Forms  has 
prompted  professors  to  resort  to  manipulative  practices  to 
obtain  higher  SOF  scores  from  students.   In  one  specific 
example,  a  professor  deliberately  failed  to  teach  a  particu- 
larly difficult  portion  of  a  course  in  an  effort  to  enhance 
his  SOF  scores.   It  was  his  announced  perception  that  higher 
SOF  scores  might  lead  to  a  step  (merit)  increase  in  salary. 
Rodin  and  Rodin's  study  found  that  students  rate  most  highly 
those  professors  from  whom  they  learned  the  least,  thus 
supporting  this  professor's  premise.  [Ref.  23] 

It  was  found  that  the  professors  of  the  NPS  departments 
that  typically  had  the  highest  mean  scores  for  SOF  Item  12 
made  a  specific  point  of  informing  their  students  that  the 
SOF's  were  used  for  pay,  promotion,  and  tenure  decisions. 
Identical  results  were  obtained  in  a  study  at  the  University 
of  Wisconsin,  Green  Bay,  where  researchers  found  "SFE  ratings 
increased  in  all  evaluation  factors  when  students  were  in- 
formed that  the  SFE  is  used  for  salary,  promotion,  and 
retention  .  .  .  ."  [Ref,  24] 
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B.   APPROACH 

The  professor  questionnaire  was  distributed  to  all 
Administrative  Sciences  professors  who  were  teaching  classes 
of  five  or  more  students,  one  questionnaire  per  class  section. 
Fifty  questionnaires  were  disseminated,  and  38  completed 
questionnaires  were  returned,  a  return  rate  of  76%.   The 
questionnaire  consisted  of  23  questions,  21  multiple  choice 
and  two  fill-in-the-blank. 

It  was  the  authors'  intention  to  poll  the  professors  on 
their  beliefs  and  opinions  regarding  the  Student  Opinion 
Forms.   The  questions  ranged  from  general  ones  about  the 
course,  the  students  in  the  class,  and  the  class  section  leader 
to  personal  questions  pertaining  to  academic  rank  and  tenure. 
There  were  also  questions  regarding  the  purpose  and  useful- 
ness of  the  SOF  and  the  amount  of  weight  SOF's  should  carry 
in  pay,  promotion,  and  tenure  decisions.   Appendix  D  contains 
a  copy  of  the  actual  questionnaire  and  the  frequencies  with 
which  the  professors  selected  each  item  option.   The  follow- 
ing is  a  synopsis  of  the  professors'  responses  to  the 
questionnaire  : 

*  Of  the  professors  who  completed  the  questionnaires, 
21%  were  teaching  Financial  Management  courses,  16% 
each  were  teaching  Manpower/Personnel  Management, 
Organizational  Effectiveness,  and  Economics/Account- 
ing courses,  11%  Information  Systems,  and  10%  other 
courses . 

*  The  majority  of  the  students  in  the  classes  whose 
professors  were  surveyed  were  in  the  Manpower/Personnel/ 
Training  Analysis,  Financial  Management,  or  Computer 
Systems  curricula. 


*  Seventy-six  percent  of  the  professors  queried  were 
not  aware  of  who  the  class  section  leader  was. 

*  Of  those  who  were  cognizant  of  the  section  leader's 
identities,  nearly  60%  anticipated  those  student  leaders 
to  fall  in  the  top  half  of  the  class,  gradewise.   Most 
of  the  professors  did  not  feel  that  the  section  leaders 
had  much  influence  over  the  other  students  in  the  class. 

*  Nearly  all  professors  (851)  thought  the  students 
liked  their  courses  at  least  fairly  well,  and  95% 
enjoyed  teaching  their  courses, 

*  Over  half  of  the  professors  surveyed  had  been  at  the 
Naval  Postgraduate  School  less  than  three  years,  and 
most  (76%)  did  not  have  tenure. 

*  Seventy-nine  percent  of  the  professors  were  teaching 
courses  that  they  considered  to  be  in  their  area  of 
professional  expertise. 

*  Well  over  half  of  the  respondents  believed  that  SOF 
forms  currently  carried  no  more  than  40%  of  the  weight 
in  pay,  promotion,  and  tenure  decisions;  13%  had  no 
idea. 

*  Seventy-four  percent  of  the  faculty  believed  that  the 
SOF  should  carry  no  more  than  40%  of  the  weight  in  pay, 
promotion,  and  tenure  decisions. 

*  The  preponderance  of  professors  did  not  feel  that 
SOF's  actually  measure  teaching  effectiveness  to  any 
great  extent,  and  they  only  found  SOF's  somewhat 
useful  in  improving  their  own  teaching  effectiveness; 
in  fact,  only  14%  of  the  faculty  believed  that  the  SOF 
measures  teaching  effectiveness  to  a  large  or  a  very 
large  extent. 


C.   METHODOLOGY 

In  addition  to  general  interest  in  the  professors' 
knowledge  and  opinions  about  the  SOF,  the  authors  also 
hoped  to  predict  each  professor's  mean  SOF  Item  12  score, 
by  course  segment,  based  on  his/her  questionnaire  responses 


It  was  the  authors'  hypothesis  that  a  combination  of  selected 
factors  from  the  questionnaire  would  have  the  capability  of 
explaining  a  large  proportion  of  the  variation  in  SOF  Item  12 
scores . 

Aggregated  data  were  obtained  from  the  Student  Opinion 
Forms  completed  by  the  students  in  the  classes  for  which  the 
professors  returned  questionnaires.   Hence,  the  authors  were 
able  to  join  the  required  data  from  both  sources  to  perform 
this  statistical  analysis. 

The  Statistical  Package  for  the  Social  Sciences  (SSPS) ,  a 
system  of  computer  programs  designed  specifically  for  use  in 
analyzing  data  from  the  social  sciences,  was  utilized  in  the 
study.   SPSS  allowed  for  the  treatment  of  missing  values,  which 
was  of  great  benefit  in  this  study  because  of  the  small  initial 
sample  size  and  the  considerable  number  of  missing  values.   Two 
of  the  questions  in  the  survey  represented  categorical  rather 
than  quantitative  variables,  so  the  SPSS  Breakdown  program  was 
used  to  replace  the  qualitative  responses  by  their  Item  12 
means  to  provide  a  usable,  numerical  format.   The  Breakdown 
program  allows  the  user  to  obtain  means,  standard  deviations, 
and  variances  of  a  numeric  dependent  variable  for  each  category 
of  an  independent  variable.   In  the  case  of  Question  1  of  the 
professor  questionnaire,  "What  is  the  general  subject  area  of 
this  course?",  Breakdown  provided  the  mean  SOF  Item  12  score 
for  all  professors  who  selected  each  offered  response.   For 
example,  the  mean  SOF  Item  12  score  for  all  professors  who 
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responded  that  they  were  teaching  Manpower/Personnel  Manage- 
ment courses  was  4.13  on  an  integer  scale  o£  0  to  5  (0 
being  "N/A",  5  being  "outstanding"). 

Once  the  questions  were  all  in  a  quantitative  format, 
15  of  the  23  were  selected  as  independent  (predictor) 
variables  for  use  in  the  statistical  analysis.   SOF  Item  12 
was  the  dependent  (criterion)  variable.   In  the  first  re- 
gression, the  15  independent  variables  selected  consisted  of 
both  "structural"  and  "subjective"  questions.   The  "struc- 
tural" questions  were  identified  as  those  over  which  the 
professor  had  little  or  no  control,  such  as  class  size, 
number  of  courses  he/she  was  teaching  that  quarter,  or 
general  subject  area  of  the  course.   The  "subjective" 
questions  were  defined  as  those  in  which  the  professors  had 
control,  such  as  knowing  who  the  class  section  leader/senior 
officer  was,  or  those  questions  that  requested  a  judgment 
or  viewpoint,  such  as  whether  the  professors  felt  they  had 
good  rapport  with  the  students  in  the  class.   The  second 
regression  involved  only  ten  "structural"  independent 
variables,  in  addition  to  SOF  Item  12,  again,  as  the 
dependent  variable. 

Table  I  lists  the  questions  from  the  professor  question- 
naire.  The  "structural"  questions  are  followed  by  (ST), 
and  the  "subjective"  questions  are  followed  by  (SUB).   The 
variables  that  were  included  in  the  regression  equations  are 
also  followed  by  an  asterisk  (*) . 
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Table  I 
Professor  Questionnaire  Items 


(Ql)   What  is  the  general  subject  area  o£  this  course? 
(ST)* 

(Q2)   What  is  the  academic  level  of  this  course?  (ST)* 

(Q3)   How  many  students  are  enrolled  in  this  class? 
(ST)* 

(Q4)   What  is  the  curriculum  number  of  the  majority  of 
this  class?  (ST)* 

(Q5)   Do  you  know  who  the  section  leader  or  senior  officer 
in  this  class  is?  (SUB)* 

(Q6)   To  what  extent  do  you  feel  the  section  leader  or 
senior  officer  has  influence  over  the  class  in 
relation  to  the  course  or  the  instructor?  (SUB) 

(Q7)   Gradewise,  what  quartile  do  you  expect  the  section 
leader  or  senior  officer  to  fall  in?  (SUB) 

(Q8)   How  well  do  you  think  the  students  generally  like 
this  course?  (SUB)* 

(Q9)   How  many  years  have  you  been  on  the  staff  at  the 
Naval  Postgraduate  School?  (ST)* 

(QIO)  What  is  your  academic  rank?  (ST) 

(Qll)  Do  you  have  tenure?  (ST)* 

(Q12)  What  step  are  you  presently  on  the  faculty  salary 
schedule?  (ST) 

(Q13)  To  what  extent  do  you  consider  this  course  to  be  in 
your  area  of  professional  expertise?  (ST)* 

(Q14)  How  many  courses  (exclusive  of  labs)  are  you 
teaching  this  quarter?  (ST)* 

(Q15)  How  many  times  have  you  taught  this  course  prior 
to  this  quarter?  (ST)* 
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(Q16)  Have  you  ever  taught  this  student  group  (plus  or 
minus  a  few  students)  before  this  quarter?  (ST)* 

(Q17)  How  well  do  you  like  teaching  this  course?  (SUB)* 

(Q18)  To  what  extent  do  you  feel  that  you  have  good 
rapport  with  this  class?  (SUB)* 

(Q19)  Do  you  tend  to  organize  and  present  this  course 
differently  according  to  the  seniority  of  the 
student  group  that  you  are  teaching  (e.g.,  first 
quarter  vs.  last  quarter  thesis  students)?  (SUB) 

(Q20)  How  much  weight  (percentage -wise)  do  you  think  the 
SOF's  presently  carry  in  pay,  promotion,  and  tenure 
decisions?  (SUB) 

(Q21)  How  much  weight  (percentage -wise)  do  you  think  the 
SOF's  should  carry  in  pay,  promotion,  and  tenure 
decisions?  (SUB) 

(Q22)  To  what  extent  do  you  feel  SOF's  actually  measure 
teaching  effectiveness?  (SUB)* 

(Q23)  How  useful  do  you  feel  the  SOF  is  in  improving  your 
teaching  effectiveness?  (SUB) 


Because  of  the  small  sample  of  38  completed  question- 
naires that  were  available  for  statistical  analysis,  a  step- 
wise regression  was  decided  upon.   This  procedure  allows 
a  researcher  to  isolate  a  small  subset  of  the  available 
independent  variables  that  will  result  in  an  optimal  predic- 
tion equation.   The  SPSS  Stepwise  Regression  program  allows 
the  researcher,  in  particular,  to  specify  three  statistical 
criteria  to  be  used  in  determining  which  predictor  variables 
will  be  selected.   The  first  parameter  stipulates  the 
maximum  number  of  predictor  variables  that  will  be  included 
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in  the  equation.   The  second  parameter  identifies  the  minimum 
F  value  (which  is  automatically  computed  to  test  for  the 
significance  of  the  regression  coefficient)  that  the  researcher 
will  accept  for  variables  that  are  to  be  entered  in  the 
regression.   The  third  stipulation  specifies  the  tolerance, 
or  one  minus  the  squared  multiple  correlation  that  a  candi- 
date independent  variable  has  with  the  other  independent 
variables  already  selected.   It  was  decided  that  a  total  of 
five  variables  would  be  selected,  with  the  default  values 
for  both  the  F-test  (.01)  and  the  tolerance  (.001)  used  so 
as  to  minimize  the  restrictions  placed  on  the  stepwise 
regression. 

Missing  data  for  the  Breakdown  analysis  was  handled  by 
eliminating  only  the  cases  in  which  the  dependent  variable 
was  missing.   (In  all  other  instances,  the  case  was  included.) 
In  the  regression  analysis,  the  default  option,  listwise 
deletion  of  missing  data,  was  used.   This  option  auto- 
matically deletes  all  cases  in  which  any  missing  values 
occur.   Thus,  all  statistics  computed  in  conjunction  with  the 
regression  program  were  based  only  on  the  remaining  cases. 
Because  of  the  numerous  missing  values  encountered,  the  small 
sample  size  used  in  this  study  was  narrowed  down  to  only 
28  cases  through  listwise  deletion. 

As  part  of  the  regression  program,  a  table  of  correla- 
tion coefficients  was  provided  to  check  for  multicollinearity 
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among  the  independent  variables.   It  is  important  to  minimize 
the  effects  of  multicollinearity  so  as  not  to  confound  the 
relative  importance  of  independent  variables  entered  in  the 
regression  equation.   With  only  a  few  exceptions,  there 
were  very  low  intercorrelations  between  the  predictor 
variables,  the  majority  falling  below  0.35. 

The  first  regression  analysis  (involving  both  "structural" 
and  "subjective"  variables)  entered  Question  4  on  step  one. 
The  F  value,  indicative  of  statistical  significance,  was 
high,  at  18.83,  with  a  very  impressive  multiple  R  of  0.65. 
Thus,  with  just  one  variable  entered,  over  421  of  the  varia- 
tion in  mean  SOF  Item  12  scores  was  explained.   Step  two 
entered  Question  8,  increasing  the  multiple  R  to  0.76. 
Question  1  was  included  in  step  three,  with  a  multiple  R  of 
0.79,  and  step  four  brought  in  Question  17,  to  raise  the 
multiple  R  to  0.82.   Although  Question  16  was  entered  in 
step  five,  the  F  value  for  this  entry  was  only  0.82. 
Because  of  the  low  statistical  significance  of  this  value, 
only  the  first  four  variables  were  used  in  constructing  the 
regression  equation. 

In  the  second  regression  analysis  (involving  only  the 
"structural"  variables).  Question  4  was,  again,  the  first 
variable  to  be  entered,  with  an  F  value  of  18.83  and  a 
multiple  R  of  0.65.   By  including  Question  3  in  the  second 
step  and  Question  9  in  the  third  step,  a  cumulative  multiple  R 
of  0.69  was  attained.   This  R  represents  48%  of  the  variation 
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in  mean  SOF  Item  12  scores  being  explained  by  the  three 
variables,  curriculum  of  students,  number  of  students  in  the 
class,  and  number  of  years  the  professor  has  been  on  the 
NPS  faculty.   The  fourth  and  fifth  variables  to  be  entered 
in  the  regression  analysis,  Questions  11  and  16,  respectively, 
were  not  included  in  the  final  regression  equation  because 
of  their  low  statistical  significance  (both  F  values  less 
than  2.0) . 

After  both  regressions  were  run,  the  corresponding  two 
regression  equations  were  constructed  using  the  unstandardized 
regression  coefficients  (B's)  and  the  constants  that  were 
computed  in  the  analysis.   The  variables  predicted  from  these 
equations  were  labeled  "PSOF"  (predicted  SOF) ,  as  these  were 
the  mean  SOF  values  that  could  be  predicted  for  each  pro- 
fessor based  on  his/her  responses  to  the  selected  questions 
from  the  survey.   Another  variable,  "RESSOF"  (residual  SOF), 
was  also  calculated  by  subtracting  each  professor's  pre- 
dicted mean  SOF  value  (PSOF)  from  his/her  actual  SOF  Item  12 
score.   RESSOF  indicates  the  amount  by  which  the  actual  mean 
SOF  Item  12  score  exceeds  or  falls  short  of  the  mean  pre- 
dicted SOF  score.   The  RESSOF  indicates  a  professor's  unique 
contribution  to  his/her  SOF  score  apart  from  what  could  have 
been  predicted  by  the  questionnaire.   Table  II  provides 
the  mean  SOF  Item  12,  PSOF,  and  RESSOF  scores  for  the  28 
cases  that  were  used  in  the  analysis.   A  positive  RESSOF 
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identifies  those  cases  in  which  the  professor  received  a 
higher  mean  SOF  Item  12  score  than  was  predicted  by  the 
regression.   Conversely,  a  negative  RESSOF  indicates  that  a 
professor  would  have  been  expected  to  receive  a  higher  mean 
SOF  Item  12  score  than  he/she  actually  did. 


Table  II 


PSOF  RESSOF 

Structural  Only 

SOF 

Instructor 

Item  12 

PSOF 

RESSOF 

1 

4.550 

4.397 

.153 

2 

4.870 

4.775 

.095 

3 

4.800 

4.775 

.025 

4 

3.750 

4.217 

-.467 

5 

3.180 

4.413 

-1.233 

6 

3.950 

4.052 

-.102 

7 

4.000 

4.108 

-.108 

8 

3.380 

3.546 

-.166 

9 

4.190 

4.084 

.106 

10 

4.320 

4.084 

.236 

11 

4.090 

4.144 

-.054 

12 

3.640 

4.074 

-.434 

13 

4.000 

3.798 

.202 

14 

3.800 

3.798 

.002 

15 

5.000 

4.832 

.168 

16 

5.000 

4.790 

.210 

17 

4.770 

4.371 

.399 

18 

4.950 

4.287 

.663 

19 

3.330 

3.858 

-.528 

20 

3.720 

3.928 

-.208 

21 

4.600 

4.238 

.362 

22 

4,170 

4.462 

-.292 

23 

4.680 

4.678 

.002 

24 

4.800 

4.497 

.303 

25 

4.060 

4.004 

.056 

26 

4.050 

4.494 

-.444 

27 

4.210 

3.756 

.454 

28 

4.030 

3.704 

.326 
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Structural  and  Subjective 


Instructor 

Item  12 

PSOF 

RESSOF 

1 

4.550 

4.470 

.080 

2 

4.870 

4.866 

.004 

3 

4.800 

4.866 

-.066 

4 

3.750 

4.025 

-.275 

5 

3.180 

3.908 

-.728 

6 

3.950 

3.822 

.128 

7 

4.000 

3.822 

.178 

8 

3.380 

3.805 

-.425 

9 

4.190 

4.237 

-.047 

10 

4.320 

4.237 

.083 

11 

4.090 

4.478 

-.388 

12 

3.640 

4.197 

-.557 

13 

4.000 

3.641 

.359 

14 

3.800 

3.641 

.159 

15 

5.000 

4.920 

.080 

16 

5.000 

5.201 

-.201 

17 

4.770 

4.369 

.401 

18 

4.950 

4.665 

.285 

19 

3.330 

3.796 

-.466 

20 

3.720 

3.796 

-.076 

21 

4.600 

4.267 

.333 

22 

4,170 

3.807 

.363 

23 

4.680 

4.741 

-.061 

24 

4.800 

4.353 

.447 

25 

4.060 

4.033 

.027 

26 

4.050 

3.960 

.090 

27 

4.210 

4.029 

,181 

28 

4.030 

3.922 

.108 

The  final  analysis  that  was  performed  on  these  new 
variables  was  a  Pearson  correlation  between  the  RESSOF  and 
the  actual  mean  SOF  Item  12  score.   This  correlation  repre- 
sents the  strength  of  the  linear  relationship  between  these 
two  variables,  with  possible  values  ranging  from  -1  for  a 
perfect  negative  correlation  to  +1  indicating  a  perfect 
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positive  correlation.   A  Pearson  correlation  of  .58  was 
obtained  for  the  analysis  in  which  both  structural  and 
subjective  variables  were  used,  and  .73  when  only  the 
structural  variables  were  entered.   These  figures  indicate 
how  much  the  mean  SOF  Item  12  scores  reflect  the  individual 
contribution  of  a  professor,  plus  error  of  measurement.   In 
the  correlation  which  includes  both  structural  and  subjective 
variables,  the  .58  signifies  that,  at  most,  341  of  the 
variance  in  the  SOF  Item  12  scores  is  attributable  to  a 
professor's  unique  contribution.   Likewise,  in  the  analysis 
of  the  solely  structural  variables,  at  most,  53%  of  the 
variance  in  the  SOF  Item  12  scores  reflects  the  professor's 
individual  qualities. 

D.   RESULTS  AND  APPLICATIONS 

The  analysis  of  the  professor  questionnaire  produced 
some  interesting  and  potentially  far-reaching  results.   The 
relative  variability  of  the  PSOF  (predicted  SOF)  variable, 
or  mean  SOF  Item  12  score  that  could  be  predicted  based  on 
the  professor's  responses  to  the  survey,  has  some  important 
implications.   The  finding  that  67%  of  the  differences  among 
professors'  mean  SOF  Item  12  scores  are  attributable  to 
factors  not  related  to  teaching  effectiveness  or  the  quality 
of  the  course  diminishes  the  credence  that  can  be  placed  in 
the  SOF  as  a  measurement  tool.   Royce  (1956)  found  in  his 
study  that  instructors  who  are  entertainers  in  class. 
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although  their  teaching  quality  may  be  mediocre,  tend  to 
receive  higher  student-faculty  evaluations  than  those  who 
are  not  so  humorous  or  personable  but  who  may  be  extremely 
effective  teachers.  [Ref.  25]   The  inclusion  of  Question  8, 
"How  well  do  you  think  the  students  generally  like  this 
course?",  in  the  second  step  of  the  first  regression  could 
be  partially  explained  by  this  tendency.   Another  potential 
contribution  to  the  strength  of  this  variable  [SOF  Item  12) 
may  be  how  enthusiastic,  involved,  and  interested  the  pro- 
fessor is  in  the  course.   Guthrie  (1954)  found  this  to  be  a 
factor  in  how  fond  the  students  were  of  particular  instruc- 
tors. [Ref.  26] 

The  inclusion  of  Question  4  (curriculum  membership  of 
the  class  majority)  in  the  first  step  of  the  regression 
implies  a  difference  in  SOF  scoring  based  on  discipline. 
Curriculum  857,  Organizational  Effectiveness,  has  the 
highest  mean  SOF  Item  12  scores  of  all  the  curricula 
surveyed  (a  total  of  23) .   Can  this  finding  be  attributed 
to  the  superiority  of  professors  in  that  discipline?   Is  it 
a  difference  in  their  teaching  approach  or  style?   Perhaps 
students  in  that  field  are,  by  nature,  more  lenient 
evaluators .   In  any  case,  it  appears  that  there  are  trends 
or  similar  tendencies  among  students  in  specific  curricula. 

Another  variable  (the  fourth)  that  was  important  to  the 
variances  in  Item  12  mean  scores,  subject  area,  is  not 
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surprising.   Certain  subject  areas  seem  to  be  universally 
more  attractive  to  students  than  others.   In  this  study, 
the  professors  teaching  Logistics/Material  Management  courses 
received  the  highest  mean  SOF  Item  12  scores,  at  4.68,  with 
instructors  in  Financial  Management  courses  following,  with 
a  mean  of  4.38.   Information  Systems  and  Economics/Accounting 
categories  received  the  lowest  mean  Item  12  ratings,  at  3.8 
and  4.0,  respectively. 

The  fifth  variable  entered  in  the  first  regression, 
"Have  you  ever  taught  this  student  group  before  this  quarter?", 
is  also  a  reasonable  factor  in  overall  professor  ratings.   The 
effect  of  this  variable  could  be  directly  or  inversely 
related  to  the  professor's  evaluation  depending  on  whether 
the  students  had  had  positive  or  negative  experiences  in 
their  previous  classes  with  the  professor. 

In  the  "structural  only"  regression,  the  number  of  students 
in  the  class  was  strongly  related  to  the  mean  SOF  Item  12 
rating  a  professor  received  (r  =  .35).   This  finding  is  in 
agreement  with  many  studies  on  this  subject.   It  has  been 
suggested  that  students  prefer  small  classes  (and  there- 
fore rate  the  professors  higher)  because  they  permit 
greater  student-teacher  interaction. 

Finally,  tenure  and  number  of  years  on  the  NPS  staff 
were  also  determinants  in  student-faculty  evaluations. 
Diverse  results  have  been  obtained  by  the  many  studies  on 
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this  subject.   Although  Heilman  and  Armentrout  [Ref.  27] 
found  no  significant  relationship  between  experience  and 
student  ratings,  Downie  [Ref.  28]  found  that  full  professors 
received  higher  ratings  than  did  instructors  in  lower 
academic  ranks.   This  was  true  especially  on  traits  such  as 
sense  of  humor,  broad  interests,  and  effective  presentation 
of  subject  matter. 

In  addition  to  the  statistical  analysis  previously  dis- 
cussed, it  was  decided  that  crosstabulations  should  be  per- 
formed on  a  few  of  the  variables,  just  as  a  matter  of 
interest.   The  amount  of  weight  an  instructor  thought  the 
SOF  should  carry  in  pay,  promotion,  and  tenure  decisions  was 
crosstabulated  by  the  academic  rank  of  the  professor.   One 
hundred  percent  of  the  professors,  associates,  and  assistants 
believed  that  less  than  40%  of  these  decisions  should  be 
based  on  SOF  score.   Forty-five  percent  of  the  adjuncts 
believed  less  than  201,  whereas  551  thought  between  41%  and 
601  of  the  weight  in  management  decisions  should  be  based 
on  SOF  ratings.   Of  the  military  instructors,  half  selected 
less  than  20%  and  half  between  61%  and  80%.   Could  it  be  that 
those  with  more  experience  with  NFS  SOF's  have  learned  (or 
developed  the   opinion)  that  they  are  not  particularly 
valid  or  reliable  performance  measures  and  should  not  be 
trusted  in  such  important  career  decisions? 

Another  crosstabulation  was  executed  utilizing  the 
variables,  "How  well  do  you  think  the  students  generally 

46 


like  this  course?"  and  "How  well  do  you  like  teaching  this 
course?"  Based  on  this  analysis,  it  appears  that  the  pro- 
fessors, in  general,  like  teaching  the  courses  more  than 
they  think  students  like  taking  them! 

The  results  of  these  analyses  can  be  used  in  various 
ways  by  NPS  faculty  and  administrators,  and  hopefully  they 
will  shed  some  light  on  the  myriad  concerns  in  regard  to  the 
use  of  SOF's.   Perhaps,  as  a  result  of  this  research,  the 
school  will  have  increased  awareness  of  the  numerous  possible 
discrepancies  and  the  many  seemingly  unrelated  factors  that 
actually  have  a  considerable  impact  on  student-professor 
evaluations.   This  study  strongly  indicates  that  great 
caution  be  exercised  when  using  the  SOF's  as  a  major  or 
exclusive  determinant  in  important  decisions  such  as  pay, 
promotion,  and  tenure. 

If  the  Student  Opinion  Forms  are  to  be  heavily  weighted 
in  significant  career  decisions,  the  research  results  reported 
here  suggest  that  a  professor's  ratings  may  be  affected  to 
a  considerable  extent  by  the  subject  area  of  the  course  and 
the  student  curriculum  group  that  he/she  is  assigned  to 
instruct,  as  well  as  the  number  of  students  in  the  class. 
Because  the  department  chairman  is  responsible  for  the 
assignment  of  professors  to  courses  and  class  sections  and 
can  also  govern  class  size,  he/she  could  have  a  substantial 
influence  on  increasing  or  decreasing  a  professor's  SOF 
scores. 
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Another  more  direct  use  of  the  professor  questionnaire 
data  would  be  to  review  the  specific  responses  of  particular 
professors  to  the  attitude  questions.   The  extent  to  which 
they  do  not  enjoy  instructing  their  courses,  have  good 
rapport  with  their  students,  or  feel  SOF's  improve  their 
own  teaching  effectiveness  may  indicate  some  attitude 
problems  that  may  be  responsible,  at  least  in  part,  for 
poor  student-professor  evaluations. 
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III.   STUDENT  QUESTIONNAIRE 

A.  MOTIVATION 

In  conjunction  with  the  interest  in  the  professors' 
viewpoints  on  SOF's  and  their  effects  and  uses,  attention  was 
also  directed  toward  the  students'  perceptions.   In  day-to- 
day conversation  with  students  at  NPS,  it  became  apparent 
that  there  was  a  great  lack  of  understanding  about  the  use 
and  importance  of  the  SOF.   Students  related  a  multitude  of 
different  experiences  they  had  encountered  during  the 
administration  of  SOF's  at  the  end  of  their  various  courses. 
There  were  numerous  comments/complaints  about  not  having 
enough  time  to  complete  the  SOF's  and  about  the  professors' 
wandering  around  the  classrooms  looking  over  students' 
shoulders.   Concern  was  also  expressed  about  how  a  student's 
grade  may  be  affected  if  the  professor  were  to  see  the  SOF 
prior  to  the  determination  of  final  course  grades.   There 
was  noticeable  apathy  among  the  students  regarding  the  wasted 
effort  expended  in  the  completion  of  the  SOF.   Many  students 
seemed  to  feel  that  no  one  really  looked  at  the  SOF's  and 
that  they  served  no  function  in  management  decisions. 

B.  APPROACH 

The  student  questionnaire  was  distributed  to  258 
Administrative  Sciences  students  from  23  different  curricula. 
They  were  completed  in  class  and  collected  by  the  researchers 
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to  attain  a  100%  return  rate.   The  survey  contained  20 
questions  which  were  answered  on  a  computer  formatted  answer 
sheet.   The  data  were  processed  and  collated  by  McGraw-Hill, 
Inc.,  who  also  provided  basic  summary  statistics. 

It  seemed  important  to  obtain  the  students'  opinions, 
attitudes,  and  understanding  of  the  Naval  Postgraduate 
School's  Student  Opinion  Form  in  an  effort  to  assess  the 
reliability  of  the  SOF  data.   The  questionnaire  items  covered 
personal  topics  such  as  age,  rank,  service,  and  commission- 
ing source.   Also  included  were  opinion  questions  regarding 
factors  that  influence  teaching  effectiveness,  as  well  as 
basic  knowledge  questions  about  the  importance  and  use  of 
the  SOF.   Appendix  E  lists  the  20  items  from  the  student 
questionnaire,  the  item  response  options,  and  the  frequencies 
with  which  the  students  responded  to  each  option.   Below 
is  a  summary  of  the  frequency  analysis: 

*  The  majority  of  students  completing  the  student 
questionnaire  were  between  the  ages  of  28  and  36. 

*  Current  rank  ranged  from  02  to  06,  the  majority 
falling  into  03  and  04--approximately  86%. 

*  Commissioning  sources  varied. 

*  The  majority  of  students  had  served  between  eight 
and  eleven  years  on  active  military  duty  (congruent 
with  rank  distribution). 

*  Completion  date  of  baccalaureate  degree  varied 
from  three  to  20  years  ago,  with  the  majority 
falling  in  the  eight  to  eleven  year  range. 

*  Approximately  70%  of  the  students  completing  the 
questionnaire  were  serving  in  the  U.S.  Navy. 
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*  Approximately  701  of  the  students  found  classroom 

layout  and  seating  arrangement  at  least  reasonably  important 
for  effective  learning. 

*  Only  101  of  the  students  surveyed  understood  that  the 
primary  purpose  and  use  of  the  SOF  was  for  pay,  promo- 
tion, and  tenure  decisions.   One-third  felt  that  the 
primary  use  of  the  SOF  was  to  provide  feedback  to  the 
professor,  only,  to  help  improve  teaching  effectiveness. 
Nearly  35%  of  the  students  thought  the  primary  SOF  use 
was  to  give  the  department  chairman  an  idea  of  the  pro- 
fessor's performance  or  popularity.   Thirteen  percent 
had  no  idea  what  the  forms  were  used  for. 

*  Fifty  percent  of  the  students  felt  that  the  SOF 
should  be  for  use  by  the  department  chairman  to  evaluate 
the  professor,  in  conjunction  with  other  performance 
measures.   Three  percent  felt  the  SOF's  should  not  be 
used  at  all. 

*  Nearly  three -fourths  of  the  students  felt  that  SOF's 
should  carry  no  more  than  401  of  the  weight  in  pay,  pro- 
motion, and  tenure  decisions. 

*  For  571  of  the  students,  the  primary  source  of  know- 
ledge concerning  the  use  of  the  SOF  was  either  an 
individual  professor  or  other  students.   Twenty-one 
percent  identified  no  source  of  information. 

*  Most  students  (26%)  felt  that  a  written  narrative 
evaluation  of  a  professor  by  each  student  would  most 
increase  teaching  effectiveness.   Twenty-four  percent 
responded  that  class  visitation  by  department  chair- 
man or  other  faculty  would  produce  the  best  results. 
Only  14%  thought  that  the  current  SOF  was  the  most 
effective  evaluation  method. 

*  Forty-two  percent  of  the  respondents  believed  that 
the  professors  never  saw  the  SOF's  before  grades  were 
determined.   Thirty-six  percent  had  no  idea. 

*  Fifty-five  percent  of  the  students  surveyed  had  always 
completed  SOF's  before  they  knew  their  final  grade. 
Forty-five  percent  had,  on  some  occasions,  already  known 
their  course  grade. 

*  Nearly  half  of  the  respondents  had  completed  SOF's 
outside  the  class  at  least  once. 
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*  Sixty-five  percent  of  those  polled  had  com- 
pleted SOF's  in  the  presence  of  the  professor. 

*  Nearly  half  the  students  had,  at  one  time  or  other, 
felt  that  they  had  not  been  given  enough  class  time 
to  complete  the  SOF. 

*  Most  students  thought  that  the  average  score  for  all 
professors  in  the  Administrative  Sciences  Department 
on  SOF  Item  12,  the  overall  rating  of  the  instructor, 
was  between  3.6  and  4.5. 

*  The  majority  of  students  believed  that  either  the 
professor  only,  or  both  the  professor  and  the  depart- 
ment chairman,  saw  the  comments  on  the  back  of  the 
SOF. 

*  Eighty  percent  of  the  students  would  have  responded 
the  same  to  the  questions  on  the  front  of  the  SOF 

if  they  had  known  that  only  the  professors  saw  the 
comments  on  the  back. 

C.   METHODOLOGY 

In  addition  to  general  interest  in  the  basic  frequencies 
of  responses  to  the  student  questionnaires,  the  researchers 
thought  that  some  other  relevant  findings  might  emerge 
from  further  statistical  analysis.   It  was  believed  that 
there  may  be  some  tendencies,  or  patterns,  to  the  way  in 
which  students  responded  to  the  survey  questions  based  on 
such  factors  as  rank,  seniority  in  the  curriculum  (i.e., 
number  of  quarters  completed),  and  branch  of  service. 

The  most  fruitful  analysis  of  the  student  questionnaire 
data  consisted  of  crosstabulations ,  or  frequency  breakouts 
of  responses  to  one  question  according  to  the  student's 
responses  to  another  question.   For  the  most  part,  in  all 
of  the  crosstabulations  performed,  no  clearcut  division 


of  responses  by  seniority,  rank,  or  branch  of  service  was 
evident.   All  groups  seemed  to  have  similar  tendencies  in 
SOF  understanding  and  beliefs,  and  they  predominantly  selected 
the  same  responses  most  frequently.   There  were,  however, 
some  interesting  findings,  which  will  be  discussed  below. 

In  the  crosstabulation  between  Question  8,  understanding 
of  the  primary  purpose  and  use  of  the  SOF,  and  number  of 
quarters  completed,  the  majority  of  all  students  (regard- 
less of  seniority)  selected  options  3  and  4.   These  were 
"feedback  to  the  professor,  only"  and  "to  give  the  department 
chairman  an  idea  of  the  professor's  performance  or  popu- 
larity," respectively.   Only  those  students  in  their  fifth  or 
sixth  quarters  seemed  to  realize  that  SOF's  play  an  important 
part  in  pay,  promotion,  and  tenure  decisions.   Those  students 
in  their  first  quarter  were,  understandably,  more  likely  to 
respond,  "I  don't  know,"  (371),  than  students  in  any  other 
quarter. 

The  crosstabulation  between  Question  11,  primary  source 
of  knowledge  concerning  the  use  of  the  SOF,  and  number  of 
quarters  completed,  again,  exhibited  no  ordinal  response 
differentiation  according  to  seniority.   As  noted  in  the 
previous  section,  "the  individual  professors"  and  "other 
students"  were  the  most  frequent  selections.   It  was  of 
considerable  interest  to  find,  however,  that  more  than  20% 
of  third-quarter  students,  nearly  25%  of  second-quarter 
students,  and  45%  of  the  first-quarter  students  had  no 
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source  of  SOF  information.  Sixth-quarter  students  were  the 
only  group  who  had  received  a  considerable  amount  of  infor- 
mation from  the  department  chairman  or  academic  associate. 

In  review  of  the  crosstabulation  of  rank  with  the 
responses  to  Question  10,  "How  much  weight  do  you  think 
the  SOF's  should  carry  in  pay,  promotion,  and  tenure  deci- 
sions?", it  was  interesting  to  note  that  officers  of  the 
rank  05  (Commanders  or  Lieutenant  Colonels)  were  the  only 
group  who  predominantly  (451)  felt  that  SOF's  should  carry 
no  weight.   The  majority  of  students  in  other  ranks  believed 
it  should  be  between  one  and  forty  percent. 

Lastly,  the  crosstabulation  of  Question  6,  branch  of 
service,  with  Question  12,  "Which  of  the  following  evaluation 
methods  do  you  feel  would  most  increase  teaching  effective- 
ness?", yielded  notable  results.   Of  all  the  services 
responding,  the  U.S.  Coast  Guard  and  the  U.S.  Army  were  the 
most  satisfied  with  the  current  SOF.   The  civilian/foreign 
military  response  group  would  be  equally  content  with 
either  "individual  student-professor  conferences"  or 
"written  narrative  evaluations."  Again,  options  3  and  4, 
"written  narrative  evaluation"  and  "section-leader  conference 
with  department  chairman,"  respectively,  were  the  most 
popular  responses  across  all  services. 
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D.   RESULTS  AND  APPLICATIONS 

The  data  that  were  obtained  from  this  study  strongly 
support  the  impressions  that  came  across  during  informal 
conversations  with  Administrative  Sciences  students.   This 
survey  revealed  that  a  common  source  of  information  regard- 
ing the  use,  purpose,  and  importance  of  SOF's  is  totally 
absent.   This,  in  addition  to  the  considerable  dispersion  of 
responses  as  to  the  understood  primary  purpose  and  use  of 
the  SOF,  implies  that  students  have  no  uniform  comprehension 
of  the  use  of  this  reporting  process.   Stevens  (1978) 
reported  that  a  total  lack  of  student  knowledge  regarding 
student-faculty  evaluations  is  not  uncommon.  [Ref.  29]   In 
his  study,  several  upper  level  students  admitted  that  they 
did  not  know  what  the  SFE  was  used  for,  but  that  they  believed 
it  served  solely  as  feedback  to  the  professors.   It  can  also 
be  seen  in  the  NPS  students'  responses  to  Questions  13  through 
17  and  Question  19  that  no  universal  standards,  guidelines, 
or  requirements  for  the  administration  of  SOF's  were  adhered 
to  by  the  professors.   It  is,  therefore,  imperative  that 
complete  information  regarding  the  SOF  become  general  know- 
ledge not  only  among  the  students  but  among  faculty  members 
as  well. 

The  fact  that  only  14%  of  the  respondents  felt  that  the 
current  SOF  was  the  most  effective  of  the  evaluation  methods 
listed  for  increasing  teaching  effectiveness  suggests  that 
this  is  a  topic  that  bears  further  review.   Since  the  majority 


o£  students  lack  faith  in  the  SOF  as  an  evaluation  tool, 
this  may  imply  that  a  lack  of  sincerity  and  genuine  effort 
is  being  exerted  in  the  completion  of  SOF's.   It  is  ludi- 
crous to  suppose  that  students  will  put  forth  a  concerted 
effort  in  honestly  evaluating  a  professor  if  they  have  no 
idea  what  the  evaluation  is  used  for,  who  sees  it,  and  the 
amount  of  weight  it  carries  in  management  decisions  - -especially 
if  they  have  no  faith  in  its  usefulness. 

Fifty  percent  of  the  students  believed  that  other 
performance  measures  should  be  used  in  conjunction  with  the 
SOF  by  the  department  chairman  to  evaluate  the  professor. 
This  finding  indicates  a  feeling  that  the  SOF  is  inadequate 
and  incomplete.   They  suggest  that  a  written  narrative 
evaluation  of  the  professor  by  each  student  would  be  the  most 
beneficial  method  to  increase  teaching  effectiveness, 
closely  followed  by  class  visitations  by  the  department 
chairman  or  other  faculty.   NFS  administrators  as  well  as 
faculty  should  seriously  consider  these  recommendations, 
as  they  may  provide  more  comprehensive  and  impartial 
evaluations . 
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IV.   ADDITIONAL  SQF  ITEMS 

A.  INTRODUCTION 

The  third  and  final  section  of  this  research  project 
involved  the  addition  of  four  questions  to  the  standard  NPS 
Student  Opinion  Form.   Since  the  SOF's  are  disseminated 
quarterly  to  all  NPS  students,  and  the  additional  four 
questions  requested  information  relating  specifically  to 
each  course  a  student  was  taking,  it  was  decided  that  the 
SOF  would  be  an  appropriate  vehicle  for  asking  these 
questions.   There  are  four  spaces  provided  on  the  SOF 
(Items  17-20)  for  additional  questions.   The  four  questions 
added  were  devised  to  reflect  relationships  of  Item  12  re- 
sponses with  students'  perceptions  of  the  course  and  of  the 
professor.   A  sample  of  approximately  560  provided  responses 
to  these  items.   Since  it  has  become  traditional  to  use 
Item  12  as  an  indicator  or  gauge  of  the  mean  responses  to 
the  other  15  items,  Pearson  correlations  were  performed 
between  the  four  additional  items  and  Item  12  only.   Table  III 
lists  the  four  additional  SOF  items. 

B.  MOTIVATION 

The  responses  the  students  provided  to  the  additional 
SOF  questions  were  expected  to  be  highly  related  to  their 
responses  on  the  original  SOF  items  (Items  1-16).   In  par- 
ticular, it  was  anticipated  that  a  student  who  was  expecting 
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Table  III 
Additional  SOF  Items 

17.  What  is  your  anticipated  grade  for  this  course? 

18.  What  is  your  total  cumulative  grade  point  average  at 
the  Naval  Postgraduate  School? 

19.  Prior  to  this  quarter,  what  were  your  expectations 
o£  this  course  in  regard  to  interest,  challenge,  and 
potential  usefulness  in  your  career? 

20.  To  what  extent  do  you  think  SOF's  are  worth  your  time 
and  effort  to  fill  out? 


a  high  grade  would  be  more  likely  to  rate  both  the  professor 
and  the  course  higher  than  those  expecting  lower  grades.   It 
was  also  predicted  that  overall  grade  point  average  (GPA) 
would  be  correlated  with  the  other  item  responses;  however, 
the  researchers  would  not  venture  to  guess  whether  this 
correlation  would  be  positive  or  negative.   It  seemed  equally 
reasonable  to  suppose  that  the  students  with  higher  GPA's 
would  tend  to  be  either  harsher  graders,  as  a  whole,  or  more 
lenient  than  those  with  lower  GPA's,   If  students  had  very 
high  expectations  of  the  course  and  were  disappointed,  then 
they  might  express  their  disappointment  with  exceptionally 
low  SOF  ratings.   It  was  assumed  that,  if  the  responses 
to  Question  20,  "To  what  extent  do  you  think  SOF's  are  worth 
your  time  and  effort  to  fill  out?",  were  predominantly  low, 
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then  the  students  were  not  spending  much  time  and  effort 
completing  the  SOF's.   If  this  were  the  case,  it  would 
significantly  decrease  the  validity  of  the  SOF  as  an  evalua- 
tion tool;  and,  therefore,  SOF  use  could  be  extremely  and 
unfairly  detrimental  to  a  professor's  career. 

C.   METHODOLOGY  AND  RESULTS 

The  first  statistical  analysis  consisted  of  the  compu- 
tation of  a  Pearson  correlation  between  each  of  the  four 
additional  questions  and  SOF  Item  12  (overall  rating  of  the 
instructor).   As  was  stated  previously,  the  Pearson  corre- 
lation measures  the  linear  relationship  between  two  variables. 
Contrary  to  expectations,  all  four  correlations  were  quite 
low.   The  correlation  coefficient  between  anticipated  grade 
(Question  17)  and  SOF  Item  12  was  .10.   This  result  implies 
that  there  is  little  linear  relationship  between  the  grade  a 
student  expects  to  receive  from  a  professor  and  the  way  he/ 
she  rates  the  professor.   Again,  there  has  been  considerable 
research  performed  in  this  area,  with  very  diverse  results. 
Both  the  Guthrie  [Ref.  30]  and  the  Cohen  and  Humphrey  [Ref.  31] 
studies  found  that  there  was  a  high  linear  relationship 
between  a  student's  expected  grade  and  the  rating  given  to 
the  instructor.   However,  Rubenstein  and  Mitchell  [Ref.  32] 
and  Russel  and  Bendig  [Ref.  33]  found  exactly  the  opposite, 
in  consonance  with  the  findings  of  this  study,  that  little 
correlation  existed.   Hence,  it  can  be  seen  that  there  is 

little  agreement  among  researchers  on  this  subject. 
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The  lowest  correlation  obtained  in  the  analysis,  and  the 
only  negative  one  (-.07),  was  between  Question  18  (cumulative 
GPA)  and  SOF  Item  12.   This  finding  discounted  the  hypothesis 
that  students  with  high  GPA's  tend  to  rate  professors 
differently  from  those  with  low  GPA's.   Supporting  this 
finding  were  the  results  of  Guthrie  (1954) ,  who  concluded 
that  students  with  higher  GPA's  do  not  rate  professors  any 
differently  from  students  with  lower  GPA's.  [Ref.  34]   An 
analysis  performed  by  the  University  of  Wisconsin--Green  Bay 
Office  for  Educational  Development  (1972)  ,  using  data  from 
the  school's  Course  Comment  Questionnaires,  also  found  no 
correlation  between  overall  GPA  and  course  evaluations. 
[Ref.  35] 

The  correlation  coefficient  for  Question  19  (prior  expec- 
tations of  the  course)  and  SOF  Item  12  was  .25,  again  indi- 
cating little  (but  certainly  notable)  correspondence.   Finally, 
the  correlation  between  Question  20  (SOF's  worth  the  time 
and  effort  to  fill  out)  and  SOF  Item  12  was  also  small,  at 
.13.   The  results  of  these  correlational  analyses  are 
encouraging  in  that  they  imply  that  the  validity  of  the  NPS 
SOF's  may  not  be  highly  contaminated  by  the  potential  biases 
represented  in  Questions  17  through  20. 

Crosstabulations  were  also  carried  out  between  Questions  17, 
18,  and  19  and  Item  12.   This  analysis  was  intended  to  show 
how  the  students  who  chose  each  option  to  these  three  questions 
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rated  their  instructors.   As  previously  discussed,  there  were 
no  definite  patterns  to  the  responses  to  Item  12  based  on 
the  options  selected  for  the  three  questions  (i.e.,  low 
correlations).   Interestingly,  of  the  eight  students  who  were 
expecting  a  C+  or  below  in  their  courses  (option  0  to  Question 
17),  six  of  them  rated  their  instructors  excellent  (option  4 
to  Item  12) .   In  each  response  category  for  all  three 
questions,  the  great  majority  of  students  rated  their  instruc- 
tors above  average,  excellent,  or  outstanding.   Also  worthy 
of  mention  is  the  fact  that  in  each  response  category  for 
questions  17,  18,  and  19,  the  greatest  number  of  students 
rated  their  prcifessors  outstanding,  followed  by  the  number 
who  responded  excellent.   This  trend  continued  in  descending 
order,  with  the  fewest  students  rating  their  professors 
poor.   These  statistics  imply  considerable  inflation  in  the 
students'  evaluations  of  NPS  professors. 

Various  other  correlations  were  computed  in  an  effort 
to  test  hypotheses  about  relationships  between  Item  12  and 
the  other  variables  on  the  SOF.   It  was  anticipated  that  the 
number  of  quarters  a  student  had  completed  at  NPS  would  have 
a  bearing  on  the  student's  overall  rating  of  an  instructor. 
The  correlation  obtained  was,  in  actuality,  very  low,  at  .07. 
This  result  reveals  virtually  no  tendency  of  overall  rating 
to  vary  with  seniority.   Another  belief  was  that  a  relation- 
ship would  exist  between  the  overall  rating  of  the  instructor 
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and  whether  the  course  he/she  was  teaching  was  required  or 
elective.   The  correlation  obtained  not  only  was  low,  but 
also  negative  (-.19).   Thus,  there  was  a  slight  tendency 
for  students  taking  a  course  as  an  elective  to  rate  instruc- 
tors more  leniently  than  other  students.   Cohen  and  Humphreys, 
in  a  1960  study,  found  evidence  to  support  this  finding  that 
teachers  of  required  courses  received  lower  ratings  than  did 
teachers  of  elective  courses.  [Ref.  36] 

There  were  also  correlations  run  between  Item  12  and  the 
other  original  items  on  the  SOF.   The  majority  of  these 
variables  were  highly  related  to  Item  12,  as  Read  [Ref.  37] 
had  shown  in  his  study,  with  the  correlations  ranging  from 
.41  to  .77.   (Question  16  was  not  considered  because  of  the 
predominance  of  courses  with  no  laboratory.)   This  high 
multicollinearity  somewhat  reduced  the  independence  of  the 
variables'  explanatory  power  in  the  regressions  reported 
below. 

Contrary  to  the  expectations  expressed  earlier,  there 
was  very  little  association  between  the  way  students 
answered  the  additional  four  SOF  items  and  their  responses 
to  the  original  16  items.   The  highest  correlation  coefficient 
obtained  was  .39,  but  the  majority  were  below  .15.   This 
result,  again,  supports  the  findings  that  expected  grade, 
GPA,  and  expectations  of  the  course  do  not  determine  the 
student's  views  of  course  and  instructor  effectiveness.   It 
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also  suggests,  in  conjunction  with  the  results  discussed 
in  the  previous  paragraph,  that  SOF  Item  12  is  a  good 
summary  of  the  other  questions. 

Several  regression  analyses  were  executed,  all  of  them 
employing  SOF  Item  12  as  the  dependent  variable.   Because  of 
numerous  missing  values,  the  automatic  listwise  deletion  of 
the  regression  program  reduced  the  sample  size  to  449  in 
these  analyses.   Initially,  only  the  four  additional  SOF 
items  (Questions  17-20)  were  regressed  on  Item  12,  for  the 
purpose  of  determining  how  much  of  the  variation  in  Item  12 
was  due  to  these  items,  as  a  group.   The  multiple  R  resulting 
from  the  inclusion  of  all  four  variables  was  0.29,  revealing 
that  a  very  small  portion  of  Item  12  variation  (only  8^) 
was  possibly  a  result  of  attitudes  or  conditions  reflected 
by  these  additional  SOF  questions. 

The  second  regression,  again  using  SOF  Item  12  as  the 
criterion  variable,  included  number  of  hours  taken  this 
quarter  (HRS) ,  number  of  quarters  completed  (QTRS) ,  whether 
the  course  was  required  or  elective  (REQELEC) ,  Questions  1-11, 
and  Questions  17-20  as  the  battery  of  predictor  variables. 
Though  unreported  here,  various  other  combinations  of  predictor 
variables  from  the  SOF  were  also  regressed  on  Item  12,  and 
comparable  results  were  obtained.   The  other  four  "overall" 
variables  (Questions  13-16)  were  not  included  in  the 
regressions . 
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In  this  (the  second)  analysis,  the  stepwise  regression 
was  selected,  limiting  the  number  of  variables  to  be  included 
to  five.   Question  9,  "The  instructor  made  this  course  a 
worthwhile  learning  experience,"  was  entered  in  step  one 
with  a  multiple  R  of  0.77  and  an  F-value  of  652.   Step  two 
included  Question  3,  "The  instructor  seemed  to  know  when 
students  did  not  understand  the  material,"  raising  the 
multiple  R  to  0.82.   Question  1,  "The  course  was  well 
organized,"  came  in  on  step  three,  increasing  the  multiple  R 
only  one  more  notch,  to  0.83,   Steps  four  and  five  included 
Question  10,  "The  instructor  stimulated  my  interest  in  the 
subject  area,"  and  Question  11,  "The  instructor  cared  about 
student  progress  and  did  his  share  in  helping  us  to  learn," 
raising  the  multiple  R,  finally,  to  0.84.   Thus,  71°^  (84^) 
of  the  diversity  in  overall  instructor  ratings  is  determined 
by  these  five  variables.   In  other  words,  if  the  students 
felt  the  course  was  a  worthwhile  learning  experience,  the 
instructor  was  sensitive  to  the  students'  understanding  of 
the  material,  the  course  had  good  organization,  the  instructor 
was  able  to  interest  the  students  in  the  subject,  and  the 
instructor  genuinely  cared  about  student  progress  and  was 
instrumental  in  helping  them  learn,  then  they  were  quite 
likely  to  consider  the  professor  an  effective  instructor. 

Finally,  response  frequencies  were  determined  for  all 
20  SOP  questions  to  obtain  an  overall  distribution  of  item 
responses  for  the  Administrative  Sciences  Department.   These 
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frequencies  are  contained  in  Appendix  F.   Below  is  a 
synopsis  of  the  most  noteworthy  findings  from  the  frequency 
analysis . 

*  The  mean  response  for  Items  1  through  15  was  3.9. 

*  The  mean  response  for  SOF  Item  12  was  4.0.   The  range 
was  from  2.6  to  5.0  (using  mean  SOF  Item  12  values  for 
each  class) . 

*  All  questions  had  a  mode  (most  frequent  response)  of 
either  4  or  5,  except  Questions  16  and  20. 

*  The  items  with  the  highest  means  were  Question  6, 

"I  felt  free  to  ask  questions,"  (mean:   4.5);  Question  5, 
"I  had  confidence  in  the  instructor's  knowledge  of  the 
subject,"  (mean:   4.4);  and  Question  7,  "The  instructor 
was  prepared  for  class,"  (mean:   4.3). 

*  Eighty-four  percent  of  the  students  were  expecting 
a  B+  or  better  in  their  courses. 

*  Eighty  percent  of  the  students  had  a  cumulative  GPA 
of  3.25  or  above. 

*  Eighty-six  percent  of  the  students  had  moderate  to 
very  high  expectations  of  their  courses. 

*  Forty-two  percent  of  the  students  felt  that  SOF ' s 
were  worth  their  time  and  effort  to  fill  out  only  to  a 
moderate  extent.   This  was  the  response  most  frequently 
selected;  and  it  was  the  cause  of  Question  20 's  having 
the  lowest  mean,  2.9,  of  all  the  SOF  items  (again, 
disregarding  Question  16) . 

As  can  be  seen,  there  is  much  valuable  analysis  that  can 
be  done  on  the  Student  Opinion  Form  data.   This  study  has 
only  scratched  the  surface  of  the  research  potential  in 
this  area.   It  is  very  difficult  to  detect,  measure,  and 
analyze  factors  that  may  come  to  bear  on  a  student's  percep- 
tions of  course  and  instructor  quality.   It  was  the  researchers' 
intention  to  delve  into  four  of  these  factors  (Questions  17-20) 
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to  determine  if  they  might  represent  strong  biases  in  the 
students'  evaluations.   The  results  obtained  in  this  brief 
part  of  the  study  were  gratifying  in  that  they  suggest  that 
NPS  students  are  able  to  minimize  the  effects  of  at  least 
some  extraneous  factors  that  may  impinge  on  their  impartial 
evaluation  of  a  professor's  effectiveness. 
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V.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.   CHARACTERISTICS  OF  SAMPLE 

The  Naval  Postgraduate  School  is  considered  to  be  a 
technical  institution  geared  toward  the  Master  of  Science 
level.   The  curricula  range  from  the  "soft"  sciences  such  as 
Administrative  Sciences  and  National  Security  Affairs  to  the 
"hard"  sciences  such  as  the  numerous  engineering  programs. 

The  students  at  NPS  are  not,  by  many  measures,  typical 
graduate  students.   The  median  age  is  between  31  and  33, 
suggesting  a  higher  maturity  level  than  the  average  graduate 
student.   The  student  body  is  comprised  of  primarily  U.S. 
military  officers,  a  considerable  number  of  foreign  military 
officers,  and  a  handful  of  civilians.   They  are  highly 
structured  and  disciplined  individuals.   Although  the  cur- 
ricula are  all  considered  extremely  rigorous,  the  highly 
motivated  and  competitive  nature  of  the  students  drives  them 
to  excel. 

The  Naval  Postgraduate  School's  professors  are  recruited 
from  some  of  the  country's  best  schools.  They  are  intrigued 
by  the  maturity,  motivation,  and  dedication  of  the  students, 
as  well  as  the  school's  heavy  emphasis  on  teaching. 

The  samples  used  in  this  research  do  not  claim  to  be 
representative  of  graduate  students  in  general  or  of 
instructor  populations  outside  the  Naval  Postgraduate  School; 
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nor  are  they  representative  of  the  entire  Naval  Postgraduate 
School. 

B.   OVERVIEW  OF  RESEARCH  RESULTS 

The  following  is  a  recapitulation  of  the  prominent 
results  obtained  in  each  study: 

1.  Professor  Questionnaire 

*  Sixty-seven  percent  of  the  variation  in  SOF  Item  12 
scores  can  be  attributed  to  curriculum  of  the  students, 
how  well  the  professor  thinks  the  students  like  the 
course,  the  subject  of  the  course,  and  how  well  the 
professor  likes  teaching  the  course. 

*  A  professor's  ratings  may  be  affected  to  a  considerable 
extent  by  the  department  chairman  in  that  he/she  can 
decide  the  subject (s)  and  students  groups  that  the 
professor  teaches  as  well  as  the  number  of  students  that 
are  in  the  class  section. 

*  The  professors  are  not  well  informed  as  to  the 
importance  of  SOF's  in  pay,  promotion,  and  tenure 
decisions . 

*  Most  professors  do  not  feel  that  SOF's  actually 
measure  teaching  effectiveness  to  a  great  extent. 

*  Over  one  half  of  the  professors  surveyed  considered 
SOF's  to  be  only  somewhat  useful  in  improving  their 
teaching  effectiveness. 

2.  Student  Questionnaire 

*  It  was  found  that  very  few  students  understood  the 
purpose  and  use  of  the  SOF,  and  its  importance  in  pay, 
promotion,  and  tenure  decisions.   They  were  also  unclear 
as  to  who  was  authorized  access  to  the  SOF  information. 

*  It  was  also  found  that  the  students  have  no  common 
reliable  source  (if  any)  of  information  regarding  the 
SOF. 

*  A  number  of  students  surveyed  felt  that  other  evalua- 
tion methods  (e.g.,  class  visitation)  should  be  used  in 
conjunction  with  the  current  SOF  to  improve  teaching 
effectiveness . 
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*  The  majority  of  students  did  not  feel  that  the  current 
SOF  was  the  best  method  for  increasing  teaching 
effectiveness . 

*  Student  responses  indicated  a  lack  of  comprehensive 
guidelines  for  SOF  administration  and  completion. 

3.   Additional  SOF  Questions 

*  A  student's  anticipated  grade  for  a  course  had  a 
very  low  correlation  with  the  overall  rating  he/she 
gave  the  professor  (SOF  Item  12). 

*  A  student's  cumulative  GPA  had  very  little  relation- 
ship with  the  overall  rating  he/she  gave  the  professor. 

*  Students  generally  tended  to  have  high  prior  expec- 
tations of  their  courses  in  regard  to  interest, 
challenge,  and  potential  usefulness  in  their  careers. 

*  The  majority  of  students  felt  that  SOF's  were 
worth  their  time  and  effort  to  complete. 

Since  this  study  was  fairly  limited  in  scope,  the 

results  should  be  considered  only  suggestive;  nevertheless, 

they  do  seem  to  complement  certain  findings  previously 

discussed. 

C.   DISCUSSION  OF  SFE  FACTORS 

Presently,  the  major  defense  for  defining  quality 
teaching  in  terms  of  high  ratings  on  the  student  evaluation 
forms  is  based  on  an  analogy  between  the  student  and  the 
consumer.   The  student  may  be  in  the  best  position  to 
evaluate  the  professor's  teaching  effectiveness  since  he/she 
is  the  main  consumer  of  the  professor's  product,  namely, 
education.   In  addition  to  this  belief,  the  ease  of  distri- 
bution to  the  students,  as  well  as  the  extent  to  which 
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student-faculty  evaluation  is  already  used  in  colleges  and 
universities  throughout  the  country,  tends  to  justify  their 
continued  use  as  a  primary  method  of  enhancing  teaching 
effectiveness. 

It  is  important  that  student  views  be  widely  solicited 
in  the  form  of  SFE,  although  no  one  suggests  that  students 
are  an  infallible  judge  of  instructor  competence.   Caution 
should  be  exercised  to  avoid  sole  dependence  of  performance 
evaluation  on  quantifiable  SFE's.   Frequently,  these  evalua- 
tions are  considered  more  important  and  reliable  than  others, 
strictly  because  they  are  quantitative  and  can  be  processed 
by  a  computer.   It  should  be  noted,  however,  that  a  compre- 
hensive evaluation  process  entails  the  observation  of 
multiple  performance  measures,  weighing  and  balancing  them 
against  each  other,  and  ranking  their  importance  in  terms  of 
the  goals  and  objectives  of  the  institution. 

For  many  years  there  has  been  a  nearly  universal  per- 
ception that  publication  output  has  been  the  primary  factor 
in  promotion  decisions.   Indeed,  there  has  been  found  to  be 
a  high  correlation  between  academic  rank  and  publication 
output  rate.   This  finding  has  been  the  subject  of  much 
criticism.   Is  it  possible  that  the  reward  system  within 
higher  education  requires  a  professor  to  "publish  or  perish"? 
Certain  observations  definitely  support  this  premise.   Higher 
status  in  graduate  schools  is  conferred  upon  accomplished 
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research-scholars  rather  than  instructor-scholars.   In 

reference  to  this  belief,  a  student  was  quoted: 

I  came  to  graduate  school  wanting  to  be  a  liberal 
arts  teacher.   I  now  want  to  do  research.   I  consider 
this  is  a  moral  decline  on  my  part  but  I  have  learned 
that  research  is  where  the  money,  the  prestige  and 
the  mobility  are.  [Ref.  38] 

At  smaller  schools,  new  faculty,  who  see  themselves  as 
temporary  employees,  place  great  emphasis  on  research  as  a 
means  of  professional  survival.   The  large  universities, 
which  do  place  primary  emphasis  on  publication,  exert  a 
tremendous  influence  on  those  schools  that  do  not.   As  a 
result  of  this  influence,  professors  aspiring  to  eminence 
generally  sense  the  pressure  to  publish,  regardless  of 
institutional  affiliation. 

Although  faculty  administrators  bear  responsibility  for 
knowledge  of  the  staff's  teaching  ability,  many  obstacles 
exist  to  hinder  this  awareness.   Administrators  typically  do 
not  have  the  time  to  personally  observe  every  professor's 
teaching  capabilities,  especially  if  the  department  is  large. 
If  the  department  chairman  is  able  to  make  one  or  two  class 
visits,  this  visitation  may  be  sufficient  to  judge  certain 
elements  of  teaching,  but  it  is  not  basis  to  make  a  compre- 
hensive evaluation.   It  cannot  be  assumed  that  elements  such 
as  number  of  classes  taught,  class  size,  or  number  of  thesis 
advisees  necessarily  correspond  to  high  quality  instruction. 
Although  the  majority  of  professors  are  uncomfortable  with 
classroom  visitations  and  resent  being  "watched  over," 
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many  administrators  consider  this  technique  a  very  important 
element  in  evaluation.   However,  the  department  chairman 
may  not  be  the  most  qualified  judge  of  effective  teaching. 
In  addition,  viewpoints  solicited  from  other  faculty  members 
within  the  department  may  be  prejudiced,  may  represent  an 
inadequate  sample,  and  frequently  may  be  merely  hearsay. 
These  inherent  factors  create  serious  weaknesses  in  a  system 
that  relies  heavily  on  teaching  evaluations,  as  traditionally 
made  by  the  department  chairman  or  other  faculty  members. 

Circumstances  beyond  the  professor's  control  may 
adversely  affect  his/her  student  ratings.   Conditions  such 
as  heavy  workload,  large  classes,  being  assigned  to  teach 
courses  that  are  not  in  his/her  area  of  professional  expertise, 
or  teaching  a  newly  designed  or  particularly  difficult  course 
may  greatly  influence  the  evaluations  a  professor  receives. 
Likewise,  characteristics  that  are  peculiar  to  the  individual 
student  may  influence  the  evaluation.   Variables  such  as  the 
student's  natural  attraction  to  the  subject  matter,  his/her 
need  to  compensate  for  poor  grades,  or  whether  the  course 
is  required  or  an  elective  may  also  have  an  effect.   It  has 
also  been  found  that  students'  evaluations  tend  to  reflect 
the  personal  and  social  characteristics  of  an  instructor,  "who 
he  is"  rather  than  "what  he  does."  [Ref.  39] 

On  occasion,  situations  occur  in  which  there  exists  a 
severe  personality  conflict  between  a  student  and  a  pro- 
fessor.  In  cases  such  as  this,  a  professor  may  receive  an 
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unjustly  low  student  rating  based  on  this  student's  hostility, 
a  rating  in  no  way  related  to  his/her  teaching  quality. 
A  few  very  low  scores  may  severely  lower  a  good  professor's 
mean  rating,  and  may  well  be  a  detriment  to  his/her  career. 

Another  point  of  concern  is  the  handling  of  "N/A"  or 
"No  Comment"  responses.   If  they  are  not  disregarded  or 
treated  as  missing  values,  they  may  inadvertently  penalize  the 
professor.   Therefore,  administrators  should  be  encouraged 
to  assess  the  total  distribution  of  responses  to  student- 
faculty  evaluation  items,  and  not  just  the  mean  overall 
score.   Unfortunately,  many  such  factors  that  are  beyond 
the  professor's  control  are  not  taken  into  consideration;  and 
the  professor  is  frequently  penalized  unfairly  because  of 
them.   As  a  result,  the  professor  may  be  forced  to  offer 
only  "safe  and  familiar"  instruction. 

Students  evaluate  a  course  based  on  their  present 
experience,  but  do  not  look  at  the  potential  benefits  that 
may  be  realized  at  a  later  date.   It  has  been  found  that 
students,  if  given  another  opportunity  later  to  rate  pro- 
fessors, would  give  them  higher  scores  now  than  they  did 
when  they  took  the  course. 

In  conclusion,  the  above  circumstances,  which  are 
inherent  in  any  educational  system,  should  be  considered 
potentially  problematic;  and,  when  they  exist,  they  should  be 
brought  to  the  attention  of  school  administrators. 
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D.   RECOMMENDATIONS 

The  following  recommendations  are  derived  from  the  results 
of  this  study,  as  well  as  from  suggestions  solicited  from 
students,  faculty,  and  administrators. 

1.  This  study  has  suggested  a  definite  need  for  a  more 
comprehensive  explanation  of  the  use  and  purpose  of  the  SOF. 
Information  regarding  the  SOF's  role  in  pay,  promotion, 

and  tenure  decisions  should  be  widely  publicized.   A 
simple  remedy  might  be  the  presentation  of  this  infor- 
mation at  a  quarterly  Superintendent's  Lecture  which  is 
mandatory  for  all  new  students.   Another  possibility 
might  be  the  dissemination  of  this  information  at  a 
lower  level,  such  as  department  or  curriculum  indoctri- 
nations or  "Welcome  Aboard"  meetings  or  printed  on  the 
SOF  itself. 

2.  Standardized  instructions  for  the  completion  of  the 
SOF  must  be  reiterated  every  quarter  to  ensure  compliance 
and  to  alleviate  the  current  serious  editing  problem. 

3.  In  order  to  collect  a  larger  and  more  representative 
sample,  questionnaires  similar  to  the  ones  used  in  this 
research  should  be  administered  to  the  entire  NFS  faculty 
and  student  body.   Additionally,  further  in-depth  analysis 
of  SOF  data  is  called  for.   There  is  a  vast  amount  of 
potential  analysis  that  has  yet  to  be  tapped  in  this  area. 

4.  Information  that  the  SOF  provides  would  be  considerably 
more  helpful  to  professors  than  currently  if  they  were 
counselled  upon  receipt  of  this  feedback  by  the  depart- 
ment chairman  or  another,  knowledgeable  faculty  member. 
With  counselling,  negative  feedback  and  criticism  can  result 
in  positive  action  rather  than  a  debilitating  emotional 
issue . 

5.  It  is  imperative  that  the  professors  be  fully  aware 
of  the  implications  of  the  ratings  received  on  the  SOF's. 
If  there  is  no  differentiation  in  the  action  taken  by  the 
administration  when  a  professor  receives  a  3.0  rating  as 
opposed  to  a  5.0,  then  there  may  be  no  incentive  for  the 
professor  to  pursue  excellence  in  teaching.   There  must 
be  a  reward  or  incentive  system  that  is  directly  related 
to  the  evaluation  system. 
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6.  It  would  be  very  beneficial  to  have  an  informal 
midquarter  evaluation  to  be  used  as  an  early  indicator 
of  the  students'  initial  impressions  of  the  quality  of 
the  professor's  teaching.   This  evaluation  would  only  be 
seen  by  the  professor,  and  would  allow  time  for  cor- 
rections and  improvements  to  be  made  during  the  same  quarter 
so  that  the  current  class  could  benefit  from  their  own 
suggestions.   A  recommended  format  for  this  midquarter 
evaluation  would  be  a  written  narrative  evaluation  from 
each  student.   This  evaluation  would  pinpoint  specific 
problems  that  may  exist,  and  would  provide  a  much 

more  personal  form  of  feedback. 

7.  Results  from  the  student  questionnaire  revealed  that 
students  prefer  the  use  of  alternative  forms  of  teacher 
evaluation  in  conjunction  with  the  current  SOF.   More 
specifically,  they  suggest  class  visitation  by  the 
department  chairman  or  other  faculty  member,  a  written 
narrative  evaluation,  and  videotaping  of  class  sessions. 
If  class  visitation  is  simply  not  feasible  because  of 
time  constraints,  then  videotaping  might  be  a  preferred 
alternative,  as  some  departments  already  possess  audio- 
visual equipment. 

8.  The  handling  of  "N/A"  and  "No  Comment"  responses 
should  be  investigated  to  ensure  that  the  professors 
are  not  being  penalized  by  them.   Since  these  responses 
are  treated  as  "0"  on  a  "0"  to  "5"  integer  scale,  then 
their  inclusion  in  the  statistical  computation  or  mean 
scores  would  considerably  decrease  the  professors' 
ratings . 

9.  The  rewriting  of  the  current  SOF  to  be  a  more 
flexible  and  comprehensive  evaluation  tool  than  it  now  is 
should  be  seriously  considered.   The  current  SOF  is 
regarded  by  some  as  too  structured  because  all  the 
administration  sees  is  the  16  questions  on  the  front; 
and  these  questions  do  not  necessarily  cover  every  facet 
of  teaching  effectiveness.   For  example,  where  is  exces- 
sive homework  or  a  professor's  condescending  attitude 
indicated  on  the  SOF?   An  evaluation  form  that  is  sub- 
divided into  different  categories  should  be  constructed. 
The  professor  would  be  evaluated  only  on  those  factors 
over  which  he/she  has  control,  and  which  are  not 
vulnerable  to  the  student's  individual  idiosyncracies . 
For  instance,  an  item  such  as  how  well  the  student 

likes  the  subject  is  a  matter  of  personal  preference 
over  which  the  professor  has  limited  control.   Addition- 
ally, the  current  heavy  reliance  on  the  professor's 
overall  rating  (SOF  Item  12)  can  be  dangerous  in  that  this 
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may  be  only  a  manifestation  of  the  student's  emotional 
reactions  to  the  entire  course.   Much  of  it  is  not  related 
to  the  professor  or  his/her  teaching,  or  more  importantly, 
to  the  amount  the  student  has  learned. 

10.  It  has  been  suggested  that  the  last  question  on  the 
SOF  read,  "Do  you  have  any  comments  that  you  would 
particularly  like  the  department  chairman  to  see?" 

If  the  student  responded  yes  to  this  question,  then  it 
would  prompt  the  department  chairman  to  read  the 
"comments"  section. 

11.  It  has  also  been  suggested  that  the  number  of 
options  for  each  SOF  item  be  reduced  from  five  to 
three  (in  addition  to  the  "N/A"  and  "No  Comments" 
options),  as  most  students  are  reticent  to  rate  a 
professor  one  or  two  (i.e.,  poor  or  fair). 

12.  Another  recommendation  is  that  two  forms  of 
teaching  evaluation  be  instituted  at  NPS,  one  strictly 
to  be  used  as  feedback  to  the  professor  (qualitative) , 
and  a  second  for  use  in  administrative  decision 
making  (quantitative) . 

13.  In  addition  to  being  quantitatively  evaluated  on 
teaching  proficiency,  a  professor  should  also  receive 
a  comparable  evaluation  for  his/her  research  and  pub- 
lication efforts. 

14.  As  a  final  note,  it  is  imperative  that  all  elements 
of  a  professor's  work- -teaching  performance,  research, 
and  publication- -be  taken  into  consideration  when 
making  administrative  decisions. 
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APPENDIX  A 
STUDENT   OPINION   FORJvl 
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5  I  had  confidence  in  the  instructor  s  knowledge  of  the  subiect ®  ®  ®  ®  ©  ® 

8.  I  felt  free  to  ask  questions ©  ®  ®  ®  O  ® 

7   The  instructor  was  prepared  (or  class ®  ®  ®  ®  ®  ® 
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APPENDIX  B 
STUDENT  COUNCIL  MEMORANDUM 

MAY  78 
STUDENT  OPINION  FORM 

The  Student  Council  would  like  you  to  know  the  following  facts 
about  the  Student  Opinion  Form  (SOF) : 

What  is  the  SOF? 

The  SOF  is  the  machine -readable  form  filled  out  by  each  student 
at  the  end  of  each  quarter  in  each  course. 

Is  the  SOF  important? 

The  SOFs  (in  summary  form)  are  used  by  the  Department  Chairmen 
to  assist  in  identifying  Faculty  members  for  pay  raises  and 
in  tenure  considerations.   I'he  bUFs  are  used  by  the  individual 
instructor  to  improve  individual  teaching  techniques  and 
improving  course  material. 

How  does  the  SOF  system  work? 

Each  student  fills  out  a  form  in  every  course.   The  forms  are 
collected  and  machine  read,  storing  the  total  data.   A 
statistical  report  is  formed  from  this  data  for  each  pro- 
fessor and  is  forwarded  to  his  Department  Chairman.   The 
Chairman  uses  this  statistical  data  to  assist  in  the 
evaluation  of  the  teaching  ability  of  each  professor. 
The  original  form  with  its  free  hand  comments  intact  is 
returned  to  the  professor  concerned. 

What  are  the  problems? 

Casual  preparation  is  a  major  problem  which  can  be  corrected  by 
the  students.   Such  seemingly  inconsequential  errors  as 
illegibility,  not  using  a  #2  pencil,  failure  to  complete 
all  blocks  or  answer  all  questions,  not  to  mention  flippant 
or  insincere  remarks,  all  degrade  the  worth  of  the  form  both 
to  instructors  and  to  the  Department  Chairman.   Insufficient 
time  for  preparation  of  the  form  is  a  problem  which  should 
be  a  matter  of  concern  both  for  section  leaders  and  pro- 
fessors.  Cooperation  is  the  key  here.   The  correct  inter- 
pretation of  the  "Quarters  Completed"  block  is  to  include 
the  present  quarter  as  one  completed. 
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What  has  been  learned  from  the  SOF  data  analyzed  to  date? 

From  the  small  amount  o£  useful  data,  the  following  trends 
are  clear: 

a.  Students  have  high  confidence  in  the  instructors' 
knowledge  of  the  subject  area. 

b.  Students  have  a  universally  low  opinion  of  the 
examinations . 

c.  There  does  not  appear  to  be  a  correlation  between 
the  grades  given  for  the  course  and  the  rating  of  the  pro- 
fessor by  the  students. 

d.  In  general,  the  professors  who  get  the  "best"  marks 
from  the  students  are  the  ones  who  teach  the  most  "popular" 
courses . 


THE  STUDENT  COUNCIL  BELIEVES  THAT  THE  STUDENT  VOICE  IN  THE 
ADMINISTRATION  OF  THE  FACULTY  (IN  THE  FORM  OF  THE  SOF)  SHOULD 
BE  SUSTAINED  AND  THAT  IT  IS  THE  STUDENT'S  RESPONSIBILITY  TO 
GIVE  THIS  IMPORTANT  FEEDBACK  VEHICLE  THE  ATTENTION  THAT  IT 
DESERVES.   IF  WE  RENDER  A  CASUAL  OPINION,  IT  WILL  COME  TO 
BE  CASUALLY  REGARDED. 


The  Student  Council  welcomes  your  comments  through  your 
Representative . 
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APPENDIX  C 
SOF  RETURN  RATE 


ADMINISTRATIVE  SCIENCES  DEPARTMENT 
SOF  RETURN  RATE  TABLES 


Q1,AY»77 

Q2,AY'77 

Q3,AY'77 

Q4,AY'77 

TOTAL  SENT 
NO.  RETURNED 

672 
505 

947 
760 

788 
488 

734 
397 

RETURN  RATE  % 

75% 

80% 

62% 

54% 

Q1,AY'78 

Q2,AY'78 

Q3,AY'78 

Q4,AY'78 

TOTAL  SENT 
NO.  RETURNED 

803 
600 

833 
613 

963 
807 

825 
627 

RETURN  RATE  % 

75% 

74% 

84% 

76% 

Q1,AY'79 

Q2,AY'79 

Q3,AY'79 

Q4,AY'79 

TOTAL  SENT 
NO.  RETURNED 

910 

727 

1023 
834 

1037 
798 

1030 
638 

RETURN  RATE  % 

80% 

82% 

77% 

62% 

Q1,AY'80 

Q2,AY'80 

Q3,AY'80 

Q4,AY»80 

TOTAL  SENT 
NO.  RETURNED 

976 

762 

1009 
664 

982 

715 

820 

532 

RETURN  RATE  % 

78% 

66% 

73% 

65% 
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Q1,AY'81 

Q2,AY'81 

Q3,AY'81 

Q4,AY'81 

TOTAL  SENT 
NO.  RETURNED 

964 
674 

774 
525 

768 
661 

707 
512 

RETURN  RATE  % 

70% 

68% 

86% 

72% 

Q1,AY'82 

Q2,AY'82 

Q3,AY'82 

Q4,AY'82 

TOTAL  SENT 
NO.  RETURNED 

853 
586 

955 

703 

937 

784 

848 
700 

RETURN  RATE  % 

69% 

74% 

84% 

83% 

Q1,AY'83 

Q2,AY'83 

Q3,AY'83 

Q4,AY'83 

TOTAL  SENT 
NO.  RETURNED 

1125 

973 

1212 
911 

1111 
986 

942 
854 

RETURN  RATE  % 

86% 

75% 

89% 

91% 

Q1,AY'84 

Q2,AY'84 

TOTAL  SENT 
NO.  RETURNED 

899 

778 

1252 
1057 

RETURN  RATE  % 

87% 

84% 
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APPENDIX  D 
PROFESSOR  SOF  QUESTIONNAIRE 

The  following  questionnaire  is  part  of  a  thesis  project  that 
will  provide  information  on  the  administration  and  use  of 
Student  Opinion  Forms.   This  questionnaire  will  be  used  in 
conjunction  with  an  associated  questionnaire  which  has  been 
distributed  to  Administrative  Sciences  students.   As  this 
questionnaire  will  be  used  strictly  for  research  purposes, 
we  request  that  you  do  not  identify  yourself  on  this  form. 

We  would  greatly  appreciate  your  objective  responses  to  the 
following  twenty-three  questions.   Please  circle  your  responses 
or  fill  in  the  blanks,  as  appropriate.   A  separate  question- 
naire should  be  completed  for  each  class  segment  that  you 
are  teaching  this  quarter. 

1.  What  is  the  general  subject  area  of  this  course? 
161    (a)  Manpower/Personnel  Management 

21^  (b)  Financial  Management 

16%  (c)  Organizational  Effectiveness 

0%  (d)  Probability/Statistics/Operations  Research 

11%  (e)  Information  Systems 

16%  (f)  Economics/Accounting 

10%  (g)  Logistics/Material  Management 

10%  (h)  Other 

2.  What  is  the  academic  level  of  this  course? 
13%    (a)  1000-2000 

.-0,    (b)  3000 
^^"    (c)  4000 

3.  How  many  students  are  enrolled  in  this  class? 
Ranged  from     5   to    46 

4.  What  is  the  curriculum  number  (e.g.,  847)  of  the  majority 
of  this  class? 

360,  365,  366,  367,  620,  815,  827,  837,  847,  and  857 

5.  Do  you  know  who  the  section  leader  or  senior  officer  in 
this  class  is? 

24%    (a)  Yes 

76%    (b)  No  (If  no,  then  go  to  question  #8.) 
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6.  To  what  extent  do  you  feel  the  section  leader  or  senior 
officer  has  influence  over  the  class  in  relation  to  the 
course  or  the  instructor? 

0%    (a)  Very  great  extent 

3%    (b)  Large  extent 
29%    (c)  Moderate  extent 
13%    (d)  Small  extent 
18%    (e)  Very  small  extent 
11%    (f)  No  extent 
26%    (g)  N/A 

7.  Gradewise,  what  quartile  do  you  expect  the  section  leader 
or  senior  officer  to  fall  in? 


24% 

(a) 

Top 

34% 

(b) 

Second 

16% 

(c) 

Third 

2% 

(d) 

Bottom 

24% 

(e) 

N/A 

How 

well  ( 

lo  you  think  th 

course? 

8% 

(a) 

Extremely  well 

32% 

(b) 

Very  well 

45% 

(c) 

Fairly  well 

13% 

(d) 

Not  very  well 

0% 

(e) 

Not  at  all 

8.   How  well  do  you  think  the  students  generally  like  this 


9.   How  many  years  have  you  been  on  the  staff  at  the 
Naval  Postgraduate  School? 
18%    (a)  Less  than  one  year 
40%    (b)  One  to  three  years 
13%    (c)  Three  to  five  years 
5%    (d)  Five  to  seven  years 
24%    (e)  Seven  or  more  years 

10.  What  is  your  academic  rank? 
18%    (a)  Professor 

16%  (b)  Associate 

21%  (c)  Assistant 

29%  (d)  Adjunct 

16%  (e)  Military  Instructor 

11.  Do  you  have  tenure? 
24%    (a)  Yes 

769-    (b)  No 

'°°        (c)  Not  applicable 
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12.  What  step  are  you  presently  on  the  faculty  salary  schedule? 
01  (a)   1-5    21%    (e)  21-25    8%    (i)  41-45 

01  (b)   6-10    2%    Cf)  26-30   11°^    ( j )  46-50 

0%  (c)  11-15   11?;    (g)  31-35    2%    (k)  51-55 

0%  (d)  16-20   13%    (h)  36-40    0%    (1)  56-60 

32%  missing 

13.  To  what  extent  do  you  consider  this  course  to  be  in  your 
area  of  professional  expertise? 

611  (a)  Very  great  extent 

181  (b)  Large  extent 

11%  (c)  Moderate  extent 

5%  (d)  Small  extent 

5%  (e)  No  extent 

14.  How  many  courses  (exclusive  of  labs)  are  you  teaching 
this  quarter? 

21%  (a)  One 

63%  (b)  Two 

16%  (c)  Three 

0%  (d)  Four  or  more 

15.  How  many  times  have  you  taught  this  course  prior  to 
this  quarter? 

24%  (a)  Zero 

25%  (b)  One 

11%  Cc)  Two 

11%  (d)  Three 

29%  (e)  Four  or  more 

16.  Have  you  ever  taught  this  student  group  (plus  or  minus 
a  few  students)  before  this  quarter? 

24%    (a)  Yes 
76%    (b)  No 

17.  How  well  do  you  like  teaching  this  course? 
37%    (a)  Extremely  well 

45%  (b)  Very  well 

13%  (c)  Fairly  well 

5%  (d)  Somewhat  well 

0%  (e)  Not  at  all 

18.  To  what  extent  do  you  feel  that  you  have  good  rapport 
with  this  class? 

34%  (a)  Very  great  extent 

45%  (b)  Large  extent 

18%  (c)  Moderate  extent 

3%  (d)  Small  extent 

0%  (e)  Very  small  extent 
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19.  Do  you  tend  to  organize  and  present  this  course 
differently  according  to  the  seniority  of  the  student 
group  that  you  are  teaching  (e.g.,  first-quarter  vs. 
last-quarter  thesis  students)? 

241    (a)  Yes 

371    (b)  No 

39%    (c)  Not  applicable 

20.  How  much  weight  (percentage-wise)  do  you  think  the 
SOF  forms  presently  carry  in  pay,  promotion,  and 
tenure  decisionsY 

29%  (a)  0-20% 

32%  (b)  21-40% 

26%  (c)  41-60% 

0%  (d)  61-80% 

0%  (e)  81-100% 

13%  (f)  No  idea 

21.  How  much  weight  (percentage-wise)  do  you  think  the  SOF 
forms  should  carry  in  pay,  promotion,  and  tenure 
decisions/ 

40%  (a)  0-20% 

34%  (b)  21-40% 

16%  (c)  41-60% 

10%  (d)  61-80% 

0%  (e)  81-100% 

22.  To  what  extent  do  you  feel  SOF's  actually  measure 
teaching  effectiveness? 

3%  (a)  Very  large  extent 

10%  (b)  Large  extent 

45%  (c)  Moderate  extent 

37%  (d)  Small  extent 

5%  (e)  Very  small  extent 

23.  How  useful  do  you  feel  the  SOF  is  in  improving  your 
teaching  effectiveness? 

3%  (a)  Extremely  useful 

13%  (b)  Very  useful 

66%  (c)  Somewhat  useful 

5%  (d)  Not  very  useful 

13%  (e)  Not  useful  at  all 


SAMPLE  SIZE  =  38 
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APPENDIX  E 
STUDENT  SQF  QUESTIONNAIRE 

The  following  questionnaire  is  part  of  a  thesis  project  that 
will  provide  information  on  the  administration  and  use  of 
Student  Opinion  Forms.   As  this  questionnaire  will  be  used 
strictly  for  research  purposes,  we  request  that  you  do  not 
identify  yourself  on  the  answer  sheet.   We  will  be  using  the 
SOF  as  the  answer  sheet  for  this  questionnaire. 

Please  complete  the  following  boxes  at  the  top  of  the  SOF 
answer  sheet: 

Cur-Specialty  Number  (e.g.,  847) 

Hrs  This  Quarter 

Qtrs  Completed  (including  this  quarter) 

Disregard  the  preprinted  SOF  question  and  answer  the  follow- 
mg  twenty  questions  in  the  appropriately  numbered  spaces 
on  the  SOF  answer  sheet.   Please  fill  in  one  response  for 
each  question. 

We  greatly  appreciate  your  time  and  effort  in  assisting  us 
in  our  research. 

1.  What  is  your  present  age? 

6%  (5)  27  or  younger 

24%  (4)  28-30 

28%  (3)  31-33 

24%  (2)  34-36 

14%  (1)  37-39 

4%  (0)  40  or  older 

2.  What  is  your  current  rank? 

0%  (5)  01 

4%  (4)  02 

45%  (3)  03 

41%  (2)  04 

8%  (1)  05 

2%  (0)  Civilian 

3.  What  is  your  commissioning  source? 

17%    (5)  One  of  the  U.S.  military  academies  or  the  Citadel 

21%    (4)  Reserve  Officer  Training  Corps  (ROTC) 

44%    (3)  Officer  Candidate  School  or  Officer  Training 

School  (OCS,  OTS)  or  NESEP  or  AQCS 
3%    (2)  Officer  Indoctrination  School  (OIS)  or 

direct  commission 
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131    (1)  Foreign  military 
2%         (0)  Not  applicable 

4.  How  many  years  have  you  served  on  full-time  active 
military  duty? 

1%  (5)   0-3 

25%  (4)   4-7 

40%  (3)   8-11 

22%  (2)  12-15 

9%  (1)  16-19 

3%  (0)  20  or  more 

5.  How  many  years  ago  did  you  receive  your  baccalaureate 
degree? 

1%  (5)  0-3 

24%  (4)  4-7 

41%  (3)  8-11 

24%  (2)  12-15 

9%  (1)  16-19 

1%  (0)  20  or  more 

6.  What  service  are  you  in? 
68%    (5)  U.S.  Navy 

8%  (4)  U.S.  Army 

0%  (3)  U.S.  Air  Force 

8%  (2)  U.S.  Marine  Corps 

4%  (1)  U.S.  Coast  Guard 

12%  (0)  Foreign  military  or  civilian 

7.  How  important  do  you  consider  classroom  layout  and  seat 
ing  arrangement  to  effective  learning? 

10%  (5)  Extremely  important 

30%  (4)  Very  important 

32%  (3)  Reasonably  important 

12%  (2)  Mildly  important 

12%  (1)  Not  very  important 

4%  (0)  Not  a  consideration 

8.  What  is  your  understanding  of  the  primary  purpose  and 
use  of  the  SOF? 

3%    (5)  For  the  superintendent  to  evaluate  the 

professor 
35%    (4)  To  give  the  department  chairman  an  idea  of 

the  professor's  performance  or  popularity 
33%    (3)  As  feedback  to  the  professor,  only,  to  help 

improve  teaching  effectiveness 
10%    (2)  For  pay,  promotion,  and  tenure  decisions 
6%    (1}  To  influence  the  decision  to  keep,  change, 

or  delete  the  course 
13%    (0)  I  have  no  idea. 
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9.   What  do  you  think  the  SOF  should  be  used  for? 

21%    (5)  As  a  general  feedbacK;  instrument  for  the  pro- 
fessor's use  only 
50%    (4)  For  use  by  the  department  chairman  to  evaluate 
the  professor,  in  conjunction  with  other 
performance  measures 
3%    (3)  To  rank  a  professor  among  other  professors 

within  the  department 
9%    (2)  For  pay,  promotion,  and  tenure  decisions 
14%    (1)  To  influence  the  decision  to  keep,  change, 
or  delete  the  course 
3%    (0)  SOF's  should  not  be  used  at  all. 

10.  How  much  weight  do  you  think  the  SOF's  should  carry 
in  pay,  promotion,  and  tenure  decisions? 

13%  (5)  0% 

34%  (4)  1-20% 

26%  (3)  21-40% 

16%  (2)  41-60% 

8%  (1)  61-80% 

3%  (0)  81-100% 

11.  What  is  the  primary  source  of  your  knowledge  concerning 
the  use  of  the  SOF? 

4%  (5)  Curricular  officer 

7%  (4)  Department  chairman/academic  associate 

34%  (3)  Individual  professor 

10%  (2)  Welcome  aboard/NPS  indoctrination  meetings 

24%  (1)  Other  students 

21%  (0)  None 

12.  Which  of  the  following  evaluation  methods  do  you  feel 
would  most  increase  teaching  effectiveness? 

14%    (5)  Individual  student  conferences 

with  the  professor 
17%    (4)  Section  leader  conference  with  department 

chairman 
26%    (3)  Written  narrative  evaluation  of  professor 

by  each  student 
24%    (2)  Class  visitation  by  department  chairman 

or  other  faculty 
5%    (1)  Video-taping  of  class 
14%    (0)  Current  SOF 

13.  Do  you  think  the  professor  sees  the  SOF's  before 
grades  are  determined? 

3%  (5)  Yes,  always 

9%  (4)  Frequently 

7%  (3)  If  he/she  requests  to  see  them 

3%  (2)  If  he/she  has  the  permission  of  the 
department  chairman 

42%  (1)  No,  never. 

36%  (0)  I  have  no  idea. 
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14.   What  percentage  of  your  time  at  NPS  have  you  completed 
SOF's  after  you  have  known  your  course  grade? 

55%  (5)  0% 

26%  (4)  1-201 

6%  (3)  21-401 

3%  (2)  41-601 

2%  (1)  61-80% 

8%  (0)  81-100% 


15.  What  percentage  of  your  time  at  NPS  have  you  completed 
SOF's  outside  of  class? 

58%  131  0% 

28%  (4)  1-20% 

2%  (3)  21-40% 

2%  (2)  41-60% 

3%  (1)  61-80% 

7%  (0)  81-100% 

16.  What  percentage  of  your  time  at  NPS  have  you  completed 
SOF's  in  the  presence  of  the  professor? 

35%  (5)  0% 

32%  (4)  1-20% 

10%  (3)  41-60% 

8%  (2)  41-60% 

7%  (1)  61-80% 

8%  (0)  81-100% 

17.  What  percentage  of  your  time  at  NPS  have  you  felt  that 
you  have  not  been  given  enough  time  to  complete  the  SOF? 
51%    (5)  0% 

24%  (4)  1-20% 

6%  C3)  21-40% 

8%  (2)  41-60% 

3%  (1)  61-80% 

8%  (0)  81-100% 

18.  What  do  you  think  the  average  score  for  all  professors 
in  the  Administrative  Sciences  Department  is  on  SOF 
Question  12,  "Overall,  I  would  rate  this  instructor.  .  ."? 

6%  (5)  4.6-5.0 

38%  (4)  4.1-4.5 

38%  (3)  3.6-4.0 

15%  (2)  3.1-3.5 

3%  (1)  2.6-3.0 

0%  (0)  2.5  or  below 

19.  Who  do  you  think  sees  the  comments  on  the  back  of  the 
SOF's? 

32%    (5)  Only  the  professor 
3%    (4)  Only  the  department  chairman 
4%    (3)  The  department  chairman  and  the  academic 
associate 
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43%    (2)  The  professor  and  the  department  chairman 

1%    (1)  The  professor  and  the  superintendent 
17%    (0)  The  professor,  academic  associate,  department 
chairman,  and  superintendent 

20.   If  you  knew  that  only  the  professor  saw  the  comments 

on  the  back  of  the  SOF's,  would  you  answer  the  questions 
on  the  front  differently  (either  higher  or  lower)? 

2%    (5)  Yes,  a  lot  higher 

4%    (4)  Yes,  a  little  higher 
80%    (3)  No,  I'd  answer  them  about  the  same. 

51    (2)  Yes,  a  little  lower 

3%    (1)  Yes,  a  lot  lower 

6%    (0)  I  don't  know. 

SAMPLE  SIZE  =  258 
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APPENDIX  F 
FREQUENCIES  OF  SOF  RESPONSES 


Ql--The  course  was  well  organized. 

Code  Freq 

0  1 

1  16 

2  51 

3  94 

4  353 

5  252 
Missing  Values  2 


Percent 

0. 

.1 

2. 

.1 

6. 

.6 

12, 

.2 

45, 

.9 

32, 

.8 

0, 

.3 

Total  769  100.0 

Mean:   4.005  Mode:   4.000 


Freq 

Percent 

0 

0.0 

21 

2.7 

61 

7.9 

123 

16.0 

319 

41.5 

244 

31.7 

1 

0.1 

769 

100.0 

Mode  : 

4.000 

Q2--Time  in  class  was  spent  effectively. 

Code 
0 
1 

2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.917 


Q3--The  instructor  seemed  to  know  when  students  didn't 
understand  the  material. 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.961 
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Freq 

Percent 

4 

0.5 

20 

2.6 

38 

4.9 

131 

17.0 

322 

41.9 

253 

32.9 

1 

0.1 

769 

100.0 

Mode  : 

4.000 

Freq 

Percent 

7 

0.9 

22 

2.9 

41 

5.3 

145 

18.9 

314 

40.8 

239 

31.1 

1 

0.1 

769 

100.0 

Mode  : 

4.000 

Q4- -Difficult  concepts  were  made  understandable 

Code 
0 
1 

2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.893 


Q5--I  had  confidence  in  the  instructor's  knowledge  of 
the  subject. 

Code 
0 
1 

2 
3 
4 
5 
Missing  Values 

Total 

Mean:   4.431 


Q6--I  felt  free  to  ask  questions 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   4.462 


Freq 

Percent 

1 

0.1 

9 

1.2 

17 

2.2 

59 

7.7 

227 

29.5 

455 

59.2 

1 

0.1 

769 

100.0 

Mode  : 

5.000 

Freq 

Percent 

0 

0.0 

8 

1.0 

17 

2.2 

44 

5.7 

242 

31.5 

456 

59.3 

2 

0.3 

769 

100.0 

Mode  : 

5.000 
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Freq 

Percent 

6 

0.8 

5 

0.7 

13 

1.7 

75 

9.8 

298 

37.6 

378 

49.2 

3 

0.4 

769 

100.0 

Mode: 

5.000 

Q7--The  instructor  was  prepared  for  the  class 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   4.311 


Q8--The  instructor's  objectives  for  the  course  have  been 
made  clear. 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   4.033 


Q9--The  instructor  made  this  course  a  worthwhile  learning 
experience . 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   4.017 


Freq 

Percent 

2 

0.3 

20 

2.6 

34 

4.4 

126 

16.4 

299 

38.9 

287 

37.3 

1 

0.1 

769 

100.0 

Mode  : 

4.000 

Freq 

Percent 

5 

0.7 

27 

3.5 

42 

5.5 

117 

15.2 

262 

34.1 

315 

41.0 

1 

0.1 

769 

100.0 

Mode  : 

5.000 

93 


Freq 

Percent 

5 

0.7 

30 

3.9 

52 

6.8 

139 

18.1 

258 

33.6 

284 

36.9 

1 

0.1 

769 

100.0 

Mode  : 

5.000 

Q10--The  instructor  stimulated  my  interest  in  the  subject 
area. 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.910 


Qll--The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn. 

Code  Freq  Percent 

0  3  0.4 

1  13  1.7 

2  31  4.0 

3  119  15.5 

4  297  38.6 

5  304                 39.5 
Missing  Values 2 0  .  3 

Total  769  100.0 

Mean:   4.094  Mode:   5.000 

Q12--0verall ,  I  would  rate  this  instructor 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.982 


Freq 

Percent 

2 

0.  3 

23 

3.0 

40 

5.2 

145 

18.9 

269 

35.0 

288 

37.5 

2 

0.3 

769 

100.0 

Mode  : 

5.000 

94 


Freq 

Percent 

1 

0.1 

38 

4.9 

60 

7.8 

226 

29.4 

290 

37.7 

152 

19.8 

2 

0.3 

769 

100.0 

Mode  : 

4.000 

Q13--0verall ,  I  would  rate  this  course 

Code 
0 
1 

2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.59  3 


Q14- -Overall ,  I  would  rate  the  textbook(s) 

Code 
0 
1 

2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.25  2 


Q15--0verall ,  I  would  rate  the  quality  of  the  exams 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.197 


Freq 

Percent 

43 

5.6 

57 

7.4 

74 

9.6 

220 

28.6 

232 

30.2 

139 

18.1 

4 

0.5 

769 

100.0 

Mode  : 

4.000 

Freq 

Percent 

83 

10.8 

32 

4.2 

50 

6.5 

209 

27.2 

263 

34.3 

125 

16.3 

7 

0.9 

769 

100.0 

Mode  : 

4.000 

95 


Q16- -Overall ,  I  would  rate  the  laboratories 

Code  Freq  Percent 

0  519  67.5 

1  21  2.7 

2  20  2.6 

3  50  6.5 

4  43  5.6 

5  57                  7.4 
Missing  Values 59 7.  7 

Total  769  100.0 

Mean:   0.941  Mode:   0.000 

Q17--What  is  your  anticipated  grade  for  this  course? 

Code  Freq  Percent 

0  8  1.0 

1  20  2.6 

2  60  7.8 

3  161  20.9 

4  198  25.7 

5  114                 14.8 
Missing  Values 208 27.0 

Total  769  100.0 

Mean:   3.538  Mode:   4.000 

Q18--What  is  your  total  cumulative  grade  point  average 
at  the  Naval  Postgraduate  School? 

Code 
0 
1 
2 
3 
4 
5 
Missing  Values 

Total 

Mean:   3.455 


Freq 

Percent 

13 

1.7 

20 

2.6 

72 

9.4 

146 

19.0 

180 

23.4 

108 

14.0 

230 

29.0 

769 

100.0 

Mode  : 

4.000 
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Q19- -Prior  to  this  quarter,  what  were  your  expectations 

o£  this  course  in  regard  to  interest,  challenge,  and 
potential  usefulness  in  your  career? 

Code  Freq  Percent 

0  33  4.3 

1  19  2.5 

2  31  4.0 

3  156  20.3 

4  200  26.0 

5  122                 15.9 
Missing  Values 208 27.0 

Total  769  100.0 

Mean:   3.492  Mode:   4.000 

Q20--TO  what  extent  do  you  think  SOF's  are  worth  your 
time  and  effort  to  fill  out? 

Code 
0 
1 

2 
3 
4 
5 
Missing  Values 

Total 

Mean:   2.909 


Freq 

Percent 

38 

4.9 

52 

6.8 

58 

7.5 

236 

30.7 

127 

16.5 

49 

6.4 

209 

27.2 

769 

100.0 

Mode  : 

3.000 
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27.  Director  of  Research  Administration,  Code  0121 
Naval  Postgraduate  School 

Monterey,  California   93943 

28.  Professor  R.  R.  Read,  Code  55Re 
Department  of  Operations  Research 
Naval  Postgraduate  School 
Monterey,  California   93943 

29.  Professor  P.  H.  Milch,  Code  55Mh 
Department  of  Operations  Research 
Naval  Postgraduate  School 
Monterey,  California   93943 

30.  Richard  Elster,  Ph.D. 

Office  of  the  Assistant  Secretary  of  the  Navy 
Room  4E788 
The  Pentagon 
Washington,  DC   20350 
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31.  LT  Charles  G.  DEITCHMAN  (OP-lllE) 
Office  of  the  Chief  of  Naval  Operations 
Navy  Department 

Washington,  DC   20350 

32.  LCDR  Brenda  M.  Gagner 
8673  Hickory  Ridge  Court 
Springfield,  Virginia   22153 

33.  LT  Patricia  A.  Chmiel 
8629  Spring  Creek  Court 
Springfield,  Virginia   22153 

34.  CAPT  J.  S.  A.  Clark,  RAE 
Personnel  Branch- -Army 
Russell  Offices 
Canberra,  ACT  2600 
AUSTRALIA 

35.  LT  Carol  A.  White 

1470  Creekside  Drive  #46 
Walnut  Creek,  California   94596 

36.  LT  Vivian  G.  Melidosian 

200  North  Pickett  Street  #1109 
Alexandria,  Virginia   22304 
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